diff --git a/content/rancher/v2.x/en/tools/pipelines/_index.md b/content/rancher/v2.x/en/tools/pipelines/_index.md index 971c664c8c4..8dc68305a7e 100644 --- a/content/rancher/v2.x/en/tools/pipelines/_index.md +++ b/content/rancher/v2.x/en/tools/pipelines/_index.md @@ -46,11 +46,15 @@ When you configure a pipeline in one of your projects, a namespace specifically The pipeline's build engine. Because project users likely won't interact with Jenkins, it's managed and locked. >**Note:** There is no option to reuse existing Jenkins deployments as the pipeline engine. + + - **Docker Registry:** Out-of-the-box, the default target for your builds in an internal Docker Registry. However, you can make configurations to push to a remote registry instead. Docker Registry is only accessible from cluster nodes and cannot be directly accessed by users. + + - **Minio:** Minio storage is used to store the logs for pipeline executions. diff --git a/content/rancher/v2.x/en/tools/pipelines/configurations/_index.md b/content/rancher/v2.x/en/tools/pipelines/configurations/_index.md index 9351303b226..41850713589 100644 --- a/content/rancher/v2.x/en/tools/pipelines/configurations/_index.md +++ b/content/rancher/v2.x/en/tools/pipelines/configurations/_index.md @@ -2,96 +2,204 @@ title: Configuring Pipelines weight: 3725 --- + +Configuring a pipeline automates the process of triggering and publishing builds. This section describes how to set up a pipeline in a production environment. + +- The [Basic Configuration](#basic-configuration) section provides sequential instruction on how to configure a functional pipeline. +- The [Advanced Configuration](#advanced-configuration) section provides instructions for configuring pipeline options. + +>**Note:** Before setting up a pipeline for a production environment, we recommend trying the [Pipeline Quick Start Guide]({{< baseurl >}}/rancher/v2.x/en/tools/pipelines/quick-start-guide). + +## Pipeline Configuration Outline + +Initial configuration of a pipeline in a production environment involves completion of several mandatory procedures. + + -Rancher Pipeline supports for integration with version control providers including GitHub and GitLab. +- [1—Configuring Persistent Data for Pipeline Components](#1—configuring-persistent-data-for-pipeline-components) +- [2—Configuring Version Control Providers](#2—configuring-version-control-providers) +- [3—Configuring Pipeline Steps](#3—configuring-pipeline-steps) +- [Advanced Configuration](#advanced-configuration) -### GitHub + +## Basic Configuration -1. Go to the project that you want to run pipelines. +To configure a functional pipeline for your project, begin by completing the mandatory basic configuration steps. -2. Under resources menu select pipeline. +### 1—Configuring Persistent Data for Pipeline Components -3. Follow the directions displayed to Setup a GitHub Application. +The internal [Docker registry]({{< baseurl >}}/rancher/v2.x/en/tools/pipelines/#reg) and the [Minio]({{< baseurl >}}/rancher/v2.x/en/tools/pipelines/#minio) store use ephemeral volumes by default. This default storage works out-of-the-box and makes testing easy, but it does not help in cases of disaster recovery. We recommend that you to configure these two deployments to persist their data. -4. From GitHub, copy the Client ID and Client Secret. Paste them into Rancher. +Complete both [A—Configuring Persistent Data for Docker Registry](#a—configuring-persistent-data-for-docker-registry) _and_ [B—Configuring Persistent Data for Minio](#b—configuring-persistent-data-for-minio). -5. For GitHub enterprise setup, enable the `Use a private github enterprise installation` option. Fill in the host address of your GitHub installation. +>**Prerequisites (for both parts A and B):** +> +>- Pipeline components are deployed in the project. +>- [Persistent volumes]({{< baseurl >}}/rancher/v2.x/en/k8s-in-rancher/volumes-and-storage/#persistent-volumes) are available in the cluster. -6. Click Authenticate. +#### A—Configuring Persistent Data for Docker Registry -### GitLab +1. From the context menu, open the project for which you're going to create a pipeline. Then select the **Workloads** tab. -1. Go to the project that you want to run pipelines. +1. Find the `docker-registry` workload and select **Ellipsis (...) > Edit**. -2. Under resources menu select pipeline. +1. Expand the **Volumes** section. Make one of the following selections from the **Add Volume** menu: -3. Follow the directions displayed to Setup a GitLab Application. + - **Add Volume > Add a new persistent volume (claim)** + - **Add Volume > Use an existing persistent volume (claim)** + +1. Complete the form that displays to choose a persistent volume for the internal Docker registry. +{{% tabs %}} -4. From GitLab, copy the Client ID and Client Secret. Paste them into Rancher. +{{% tab "Add a new persistent volume" %}} +
+1. Enter a **Name** for the volume claim. -5. For private GitLab setup, enable the `Use a private gitlab enterprise installation` option. Fill in the host address of your GitLab installation. +1. Select a volume claim **Source**: -6. Click Authenticate. + - If you select **Use a Storage Class to provision a new persistent volume**, select a [Storage Class]({{< baseurl >}}/rancher/v2.x/en/k8s-in-rancher/volumes-and-storage/#storage-classes) and enter a **Capacity**. ->**Note:** If you use GitLab and your rancher setup is in a local network, make sure to enable ‘Allow requests to the local network from hooks and services’ option in GitLab admin settings. + - If you select **Use an existing persistent volume**, choose a **Persistent Volume** from the drop-down. +1. From the **Customize** section, choose the read/write access for the volume. -## Executor Quota +1. Click **Define**. -Executor quota decides how many builds can be running simultaneously in the project. If the number of triggered builds exceeds the quota, excess builds will be kept in queueing state until there is a vacant slot for it. By default, the quota is 2. +{{% /tab %}} -To change that, +{{% tab "Use an existing persistent volume" %}} +
+1. Enter a **Name** for the volume claim. -1. Go to the project as a project owner. +1. Choose a **Persistent Volume Claim** from the drop-down. -2. Under resources menu select pipeline. +1. From the **Customize** section, choose the read/write access for the volume. -3. Under `The maximum number of pipeline executors` setting click the button to change the quota. A number less than or equal to zero stands for unlimited quota. +1. Click **Define**. + +{{% /tab %}} + +{{% /tabs %}} + +1. From the **Mount Point** field, enter `/var/lib/registry`, which is the data storage path inside the Docker registry container. + +1. Click **Upgrade**. + +#### B—Configuring Persistent Data for Minio -## Pipeline Trigger +1. From the **Workloads** tab, find the `minio` workload and select **Ellipsis (...) > Edit**. -When a repository is enabled, the webhook is automatically set in version control system. By default, pipeline trigger for "push" event is enabled. +1. Expand the **Volumes** section. Make one of the following selections from the **Add Volume** menu: -To change that, + - **Add Volume > Add a new persistent volume (claim)** + - **Add Volume > Use an existing persistent volume (claim)** + +1. Complete the form that displays to choose a persistent volume for the internal Docker registry. +{{% tabs %}} -1. Go to the pipeline tab of a project. +{{% tab "Add a new persistent volume" %}} +
+1. Enter a **Name** for the volume claim. -2. For an enabled repository, click and expand the action button on the right. +1. Select a volume claim **Source**: -3. Click `Settting` + - If you select **Use a Storage Class to provision a new persistent volume**, select a [Storage Class]({{< baseurl >}}/rancher/v2.x/en/k8s-in-rancher/volumes-and-storage/#storage-classes) and enter a **Capacity**. -4. On the pop-up window, toggle trigger for "Push", "Push Request" and "Tag" events according to your need. + - If you select **Use an existing persistent volume**, choose a **Persistent Volume** from the drop-down. +1. From the **Customize** section, choose the read/write access for the volume. -5. Click `Save` +1. Click **Define**. + +{{% /tab %}} + +{{% tab "Use an existing persistent volume" %}} +
+1. Enter a **Name** for the volume claim. + +1. Choose a **Persistent Volume Claim** from the drop-down. + +1. From the **Customize** section, choose the read/write access for the volume. + +1. Click **Define**. + +{{% /tab %}} + +{{% /tabs %}} + +1. From the **Mount Point** field, enter `/data`, which is the data storage path inside the Minio container. + +1. Click **Upgrade**. -## Pipeline Steps + +### 2—Configuring Version Control Providers + +Rancher Pipeline supports integration with version control providers GitHub and GitLab. + +{{% tabs %}} +{{% tab "GitHub" %}} +1. From the context menu, open the project for which you're configuring a pipeline. + +1. From the main menu, select **Resources > Pipelines**. + +1. Follow the directions displayed to setup a GitHub application. + + ![GitHub Pipeline Instructions]({{< baseurl >}}/img/rancher/github-pipeline.png) + +1. From GitHub, copy the **Client ID** and **Client Secret**. Paste them into Rancher. + +1. If you're using GitHub for enterprise setup, select **Use a private github enterprise installation**. Enter the host address of your GitHub installation. + +1. Click **Authenticate**. +{{% /tab %}} +{{% tab "GitLab" %}} +1. From the context menu, open the project for which you're configuring a pipeline. + +1. From the main menu, select **Resources > Pipelines**. + +1. Follow the directions displayed to setup a GitLab application. + + ![GitLab Pipeline Instructions]({{< baseurl >}}/img/rancher/gitlab-pipeline.png) + +1. From GitLab, copy the **Application ID** and **Secret**. Paste them into Rancher. + +1. If you're using GitLab for enterprise setup, select **Use a private gitlab enterprise installation**. Enter the host address of your GitLab installation. + +1. Click **Authenticate**. + +>**Note:** If you use GitLab and your Rancher setup is in a local network, enable the **Allow requests to the local network from hooks and services** option in GitLab admin settings. +{{% /tab %}} +{{% /tabs %}} + + + +### 3—Configuring Pipeline Steps For convenience, there are multiple built-in step types for dedicated tasks. -### Clone step +#### Clone Step Type The first step is preserved to be a cloning step to check out the source code. -### Run Script step +#### Run Script Step Type `Run Script` step is to execute arbitrary commands inside a specified container. You can use it to build, test and do more, given whatever utilities the base image provides. -To add a `Run Script` step, - +{{% tabs %}} +{{% tab "By UI" %}} 1. Go to EditConfig view. 2. Click `Add a Step` in a stage. 3. Choose `Run Script` in `Step Type` option, fill in an image and the commands in the form. You can optionally set environment variables and conditions by expanding advanced options. -4. Click `Add` - -Or in yaml format: +4. Click `Add` +{{% /tab %}} +{{% tab "By YAML" %}} ```yaml # example @@ -101,30 +209,31 @@ stages: - runScriptConfig: image: golang shellScript: go build -``` +``` +{{% /tab %}} +{{% /tabs %}} -### Publish Image step +#### Build and Publish Image Step `Publish Image` step is to build and publish a docker image. To do that a Dockerfile is required to be present in the source code repository. -To add a `Publish Image` step, - +{{% tabs %}} +{{% tab "By UI" %}} 1. Go to EditConfig view. -2. Click `Add a Step` in a stage. +1. Click `Add a Step` in a stage. 1. Choose `Build and Publish Image` in `Step Type` option. -2. Fill in the configuration form: +1. Fill in the configuration form: - `Dockerfile Path` is the relative path to the Dockerfile in the source code repo. It is `./Dockerfile` as default and suits the case when you put the Dockerfile in the root directory. You can, for example, set it to `./path/to/myDockerfile` in other cases. - `Image Name` is the image name in 'name:tag' format. It doesn't need to contain the registry address. For example, to build `example.com/repo/my-image:dev`, input `repo/my-image:dev` here. - `Push image to remote repository` option is to tell what registry to publish the built image. If it is disabled, the built image will be pushed to the internal registry. Or you can enable it and choose a registry from registry list of the project as the publish target. - `Build Context` configuration is in advanced options. It is by default `.` representing the root directory of the source code. For more details please refer to [docs for docker build](https://docs.docker.com/engine/reference/commandline/build/) -3. Click `Add` - -Or in yaml format: - +1. Click `Add` +{{% /tab %}} +{{% tab "By YAML" %}} ```yaml # example stages: @@ -137,23 +246,33 @@ stages: pushRemote: true registry: example.com ``` +You can use specific arguments for docker daemon and the build. They are not exposed in the UI but are available in pipeline yaml format, as indicated in the above example. Available variables includes: -### Deploy YAML step +Variable Name | Description +------------------------|------------------------------------------------------------ +PLUGIN_DRY_RUN | Disable docker push +PLUGIN_DEBUG | Docker daemon executes in debug mode +PLUGIN_MIRROR | Docker daemon registry mirror +PLUGIN_INSECURE | Docker daemon allows insecure registries +PLUGIN_BUILD_ARGS | Docker build args, a comma separated list +{{% /tab %}} +{{% /tabs %}} + +#### Deploy YAML Step Type `Deploy YAML` step is to deploy arbitrary Kubernetes resources inside the project. To do that a Kubernetes manifest file is required to be present in the source code repository. Pipeline variable substitution is supported in the manifest file. For available variables please refer to [variable reference](https://TODO). -To add a `Deploy YAML` step, - +{{% tabs %}} +{{% tab "By UI" %}} 1. Go to EditConfig view. 2. Click `Add a Step` in a stage. 3. Choose `Deploy YAML` in `Step Type` option, fill in `YAML Path` which is the path to the manifest file in the source code. -4. Click `Add` - -Or in yaml format: - +4. Click `Add` +{{% /tab %}} +{{% tab "By YAML" %}} ```yaml # example stages: @@ -161,32 +280,100 @@ stages: steps: - applyYamlConfig: path: ./deployment.yaml -``` +``` +{{% /tab %}} +{{% /tabs %}} -## Conditions -You can set conditions for a pipeline/stage/step. If all conditions evaluate to true the pipeline/stage/step is executed, otherwise it is skipped. When a stage/step is skipped, it is considered as SUCCESS and follow-up stages/steps continue to run. Wildcard character(`*`) expansion is supported in conditions. -To set branch conditions of a pipeline, -1. Go to EditConfig view. +## Advanced Configuration -2. Click Show advanced options for the pipeline configuration. +### Configuring the Executor Quota -3. Click Add Rule button in Trigger Rules to set rules for branch conditions. +The executor quota decides how many builds can run simultaneously in the project. If the number of triggered builds exceeds the quota, subsequent builds will queue until a vacancy opens. By default, the quota is `2`, but you can change it. -For a stage/step, +1. From the context menu, open the project for which you've configured a pipeline. -1. Click on a stage or a step. +1. From the main menu, select **Resources > Pipelines**. -2. Click Show advanced options. +1. From `The maximum number of pipeline executors` increment the **Scale** up or down to change the quota. A value of `0` or less removes the quota limit. -3. Click Add Rule button in Trigger Rules. -4. Choose Branch or Event in Type drop-down, input Value for branch conditions, or choose Value for event conditions. +### Configuring Pipeline Trigger Rules -Or in yaml format: +When a repository is enabled, a webhook for it is automatically set in the version control system. By default, the project pipeline is triggered by a push event to a specific repository, but you can add (or change) events that trigger a build, such as a pull request or a tagging. +Trigger rules come in two types: + +- **Run this when:** + + This type of rule starts the pipeline, stage, or step when a trigger explicitly occurs. + +- **Do Not Run this when:** + + If all conditions evaluate to true the pipeline/stage/step is executed, otherwise it is skipped. When a stage/step is skipped, it is considered as SUCCESS and follow-up stages/steps continue to run. Wildcard character(`*`) expansion is supported in conditions. + + +{{% tabs %}} +{{% tab "Pipeline Trigger" %}} +1. From the context menu, open the project for which you've configured a pipeline. Then select the **Pipelines** tab. + +1. From the pipeline for which you want to edit build triggers, select **Ellipsis (...) > Edit Config**. + +1. Click **Show advanced options**. + +1. From **Trigger Rules**, configure rules to run or skip the pipeline, stage, or step. + + 1. Click **Add Rule**. In the **Value** field, enter the name of the branch that triggers the pipeline. + + 1. **Optional:** Add more branches that trigger a build. +{{% /tab %}} +{{% tab "Stage Trigger" %}} +1. From the context menu, open the project for which you've configured a pipeline. Then select the **Pipelines** tab. + +1. From the pipeline for which you want to edit triggers, select **Ellipsis (...) > Edit Config**. + +1. From the pipeline stage that you want to configure a trigger for, click the **Edit** icon. + +1. Click **Show advanced options**. + +1. Add one or more trigger rules. + + 1. Click **Add Rule**. + + 1. Choose the **Type** that triggers the stage. + + | Type | Value | + | ------ | -------------------------------------------------------------------- | + | Branch | The name of the branch that triggers the stage. | + | Event | The type of event that triggers the stage (Push, Pull Request, Tag). | + +1. Click **Save**. +{{% /tab %}} +{{% tab "Step Trigger" %}} +1. From the context menu, open the project for which you've configured a pipeline. Then select the **Pipelines** tab. + +1. From the pipeline for which you want to edit triggers, select **Ellipsis (...) > Edit Config**. + +1. From the pipeline step that you want to configure a trigger for, click the **Edit** icon. + +1. Click **Show advanced options**. + +1. Add one or more trigger rules. + + 1. Click **Add Rule**. + + 1. Choose the **Type** that triggers the step. + + | Type | Value | + | ------ | -------------------------------------------------------------------- | + | Branch | The name of the branch that triggers the stage. | + | Event | The type of event that triggers the stage (Push, Pull Request, Tag). | + +1. Click **Save**. +{{% /tab %}} +{{% tab "Do Not Run YAML" %}} ```yaml # example stages: @@ -209,8 +396,10 @@ branch: include: [ master, feature/*] exlclude: [ dev ] ``` +{{% /tab %}} +{{% /tabs %}} -## Timeout +### Timeout By default, A pipeline execution has 60 minutes timeout. To change that, @@ -233,7 +422,7 @@ stages: timeout: 30 ``` -## Environment Variables +### Environment Variables You can set environment variables to individual steps. @@ -262,12 +451,14 @@ stages: SECOND_KEY: VALUE2 ``` -## Secrets +### Secrets You can use Kubernetes secrets from the project in your pipeline steps. >**Note:** Secret injection is disabled on pull request events. +{{% tabs %}} +{{% tab "By UI" %}} 1. Go to the project page. 2. Under Resources menu, click Secrets. @@ -282,10 +473,9 @@ You can use Kubernetes secrets from the project in your pipeline steps. 7. Choose the secret created above and choose the key. You can optionally input an alias. -8. Now the secret is injected as environment variables for this step. - -Or in yaml format: - +8. Now the secret is injected as environment variables for this step. +{{% /tab %}} +{{% tab "By YAML" %}} ```yaml # example stages: @@ -299,4 +489,6 @@ stages: - sourceName: my-secret sourceKey: secret-key targetKey: ALIAS_ENV -``` \ No newline at end of file +``` +{{% /tab %}} +{{% /tabs %}} diff --git a/src/img/rancher/github-pipeline.png b/src/img/rancher/github-pipeline.png new file mode 100644 index 00000000000..d80b392fdd9 Binary files /dev/null and b/src/img/rancher/github-pipeline.png differ diff --git a/src/img/rancher/gitlab-pipeline.png b/src/img/rancher/gitlab-pipeline.png new file mode 100644 index 00000000000..e19f784a76c Binary files /dev/null and b/src/img/rancher/gitlab-pipeline.png differ