From 576bf66e03016bdb81ef1b24dab8b80980d23049 Mon Sep 17 00:00:00 2001 From: Jack Baldry Date: Tue, 22 Apr 2025 17:53:41 +0100 Subject: [PATCH] Add Observability as Code documentation (#104301) Co-authored-by: Kim Nylander Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com> --- docs/sources/observability-as-code/_index.md | 79 ++++++ .../observability-as-code/get-started.md | 78 ++++++ .../provision-resources/_index.md | 77 ++++++ .../provision-resources/file-path-setup.md | 162 ++++++++++++ .../provision-resources/git-sync-setup.md | 238 ++++++++++++++++++ .../provision-resources/intro-git-sync.md | 88 +++++++ .../provisioned-dashboards.md | 137 ++++++++++ .../provision-resources/use-git-sync.md | 84 +++++++ 8 files changed, 943 insertions(+) create mode 100644 docs/sources/observability-as-code/_index.md create mode 100644 docs/sources/observability-as-code/get-started.md create mode 100644 docs/sources/observability-as-code/provision-resources/_index.md create mode 100644 docs/sources/observability-as-code/provision-resources/file-path-setup.md create mode 100644 docs/sources/observability-as-code/provision-resources/git-sync-setup.md create mode 100644 docs/sources/observability-as-code/provision-resources/intro-git-sync.md create mode 100644 docs/sources/observability-as-code/provision-resources/provisioned-dashboards.md create mode 100644 docs/sources/observability-as-code/provision-resources/use-git-sync.md diff --git a/docs/sources/observability-as-code/_index.md b/docs/sources/observability-as-code/_index.md new file mode 100644 index 00000000000..3f92ff84655 --- /dev/null +++ b/docs/sources/observability-as-code/_index.md @@ -0,0 +1,79 @@ +--- +_build: + list: false +noindex: true +cascade: + noindex: true +description: Overview of Observability as Code including description, key features, and explanation of benefits. +keywords: + - observability + - configuration + - as code + - dashboards + - git integration + - git sync + - github +labels: + products: + - enterprise + - oss +title: Observability as Code +weight: 100 +--- + +# Observability as Code + +Observability as Code lets you apply code management best practices to your observability resources. +Using Observability as Code, you can version, automate, and scale Grafana configurations, including dashboards and observability workflows. +By representing Grafana resources as code, you can integrate them into existing infrastructure-as-code workflows and apply standard development practices. + +Observability as Code provides more control over configuration. Instead of manually configuring dashboards or settings through the Grafana UI, you can: + +- **Write configurations in code:** Define dashboards in JSON or other supported formats. +- **Sync your Grafana setup to GitHub:** Track changes, collaborate, and roll back updates using Git and GitHub, or other remote sources. +- **Automate with CI/CD:** Integrate Grafana directly into your development and deployment pipelines. +- **Standardize workflows:** Ensure consistency across your teams by using repeatable, codified processes for managing Grafana resources. + +{{< section depth=5 >}} + + diff --git a/docs/sources/observability-as-code/get-started.md b/docs/sources/observability-as-code/get-started.md new file mode 100644 index 00000000000..6e7c53d772c --- /dev/null +++ b/docs/sources/observability-as-code/get-started.md @@ -0,0 +1,78 @@ +--- +description: Get started with Observability as Code by exploring the documentation, libraries, and tools available for as-code practices. +keywords: + - configuration + - as code + - as-code + - dashboards + - Git Sync + - Git +labels: + products: + - enterprise + - oss +title: Get started with Observability as Code +weight: 100 +--- + +# Get started with Observability as Code + +Simply put, with Observability as Code, you can manage Grafana resources. +You can write code that describes what you want the dashboard to do, rather than manipulate it via the UI. + +Observability as Code lets you manage dashboards, resources, and configurations programmatically, leveraging powerful tools for automation and standardization. + +## Get started with Observability as Code + + + +1. [**Set up Git Sync**](https://grafana.com/docs/grafana//observability-as-code/provision-resources/git-sync-setup/) + + - Configure Git repositories to store your dashboard JSON files. + - Understand best practices for version control, including collaboration through pull requests and rollbacks. + - Edit your JSON files in GitHub and then sync with Grafana. + +1. [**Manage dashboard deployments from GitHub**](https://grafana.com/docs/grafana//observability-as-code/provision-resources/use-git-sync/) + + - Integrate dashboards into CI/CD pipelines using tools like GitHub Actions. + - Leverage provisioning features in Grafana to automate updates and deployment of dashboards. + + +## Explore additional Observability as Code tools + +- [**Crossplane:**](https://github.com/grafana/crossplane-provider-grafana) Manage Grafana resources using Kubernetes manifests with the Grafana Crossplane provider. +- [**Grafonnet:**](https://github.com/grafana/grafonnet) Grafonnet is a Jsonnet library for generating Grafana dashboard JSON definitions programmatically. It is currently in the process of being deprecated. +- [**Grizzly:**](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/grizzly/dashboards-folders-datasources/) Grizzly is a command-line tool that simplifies managing Grafana resources using Kubernetes-inspired YAML syntax. It is currently in the process of being deprecated. diff --git a/docs/sources/observability-as-code/provision-resources/_index.md b/docs/sources/observability-as-code/provision-resources/_index.md new file mode 100644 index 00000000000..65785d437df --- /dev/null +++ b/docs/sources/observability-as-code/provision-resources/_index.md @@ -0,0 +1,77 @@ +--- +description: Learn about how to provision resource using Git Sync and local file provisioning administration. +keywords: + - observability + - configuration + - as code + - git integration + - git sync + - github +labels: + products: + - enterprise + - oss +title: Provision resources and sync dashboards +weight: 100 +--- + +# Provision resources and sync dashboards + +{{< admonition type="caution" >}} +Provisioning is an [experimental feature](https://grafana.com/docs/release-life-cycle/) introduced in Grafana v12 for open source and Enterprise editions. Engineering and on-call support is not available. Documentation is either limited or not provided outside of code comments. No SLA is provided. This feature isn't available in Grafana Cloud. +{{< /admonition >}} + +{{< section depth="5" >}} + +
+ +Using Provisioning, you can configure how to store your dashboard JSON files in either GitHub repositories using Git Sync or a local path. + +Of the two experimental options, Git Sync is the recommended method for provisioning your dashboards. You can synchronize any new dashboards and changes to existing dashboards to your configured GitHub repository. +If you push a change in the repository, those changes are mirrored in your Grafana instance. +For more information on configuring Git Sync, refer to [Set up Git Sync](https://grafana.com/docs/grafana//observability-as-code/provision-resources/git-sync-setup). + +Refer to [Set up file provisioning](https://grafana.com/docs/grafana//observability-as-code/provision-resources/file-path-setup/) to learn more about the version of local file provisioning in Grafana 12. + +## Provisioned folders and connections + +Dashboards and folders saved to the local path are referred to as "provisioned" resources and are labeled as such in the Grafana UI. + +Dashboards saved in your GitHub repository or local folder configured appear in a provisioned folder in Grafana. + +You can set a single folder, or multiple folders to a different repository, with up to 10 connections. Alternatively, your entire Grafana instance can be the provisioned folder. + +## How it works + +A user decides to update a provisioned dashboard that is either stored within a GitHub repository (Git Sync workflow) or in a local file (local file workflow). + +### Git Sync workflow + +Resources provisioned with Git Sync can be modified from within the Grafana UI or within the GitHub repository. +Changes made in either the repository or the Grafana UI are bidirectional. + +For example, when a user updates dashboards within the Grafana UI, they choose **Save** to preserve the changes. +Grafana notifies them that the dashboard is provisioned in a GitHub repository. +They choose how to preserve their changes: either saved directly to a branch or pushed to a new branch using a pull request in GitHub. +If they chose a new branch, then they open the pull request and follow their normal workflow. + +Grafana polls GitHub at a regular interval. +The connection is established using a personal access token for authorization. +With the webhooks feature enabled, repository notifications appear almost immediately. +Without webhooks, Grafana polls for changes at the specified interval. +The default polling interval is 60 seconds. + +Any changes made in the provisioned files stored in the GitHub repository are reflected in the Grafana database. +The Grafana UI reads the database and updates the UI to reflect these changes. + +### Local file workflow + +In the local file workflow, all provisioned resources are changed in the local files. +The user can't use the Grafana UI to edit or delete provisioned resources. + +Any changes made in the provisioned files are reflected in the Grafana database. +The Grafana UI reads the database and updates the UI to reflect these changes. + +## Explore provisioning + +{{< section withDescriptions="true" depth="5" >}} diff --git a/docs/sources/observability-as-code/provision-resources/file-path-setup.md b/docs/sources/observability-as-code/provision-resources/file-path-setup.md new file mode 100644 index 00000000000..9c8e8de5896 --- /dev/null +++ b/docs/sources/observability-as-code/provision-resources/file-path-setup.md @@ -0,0 +1,162 @@ +--- +description: Instructions for setting up file provisioning with a local path. +keywords: + - as code + - as-code + - file provisioning + - local path +labels: + products: + - enterprise + - oss +title: Set up file provisioning +weight: 200 +--- + +# Set up file provisioning + +{{< admonition type="note" >}} +Local file provisioning is an [experimental feature](https://grafana.com/docs/release-life-cycle/) introduced in Grafana v12 for open source and Enterprise editions. Engineering and on-call support is not available. Documentation is either limited or not provided outside of code comments. No SLA is provided. Enable the `provisioning` and `kubernetesDashboards` feature toggles in Grafana to use this feature. This feature isn't available in Grafana Cloud. +{{< /admonition >}} + +- [Provision resources and sync dashboards](/docs/grafana//observability-as-code/provision-resources/) + - [Git Sync](/docs/grafana//observability-as-code/provision-resources/intro-git-sync/) + - [Set up Git Sync](/docs/grafana//observability-as-code/provision-resources/git-sync-setup/) + - [Set up file provisioning](/docs/grafana//observability-as-code/provision-resources/file-path-setup/) + - [Work with provisioned dashboards](/docs/grafana//observability-as-code/provision-resources/provisioned-dashboards/) + - [Manage provisioned repositories with Git Sync](/docs/grafana/observability-as-code/provision-resources/use-git-sync/) + +
+ +File provisioning in Grafana lets you include resources, including folders and dashboard JSON files, that are stored in a local file system. + +This page explains how to set up local file provisioning. + +The local path mount is referred to as a repository. + +Using the local path lets you also use it with a tool like `fuse`, allowing you to mount S3 buckets as local paths. You can also use tools like `restic` to automatically back up your dashboards to your preferred backup storage solution. + +To set up file sync with local with local files, you need to: + +1. Enable feature toggles and paths in Grafana configuration file (first time set up). +1. Set the local path. +1. Choose what content to sync with Grafana. + +## New file provisioning capabilities + +Local file provisioning using **Administration** > **Provisioning** will eventually replace the traditional methods Grafana has used for referencing local file systems for dashboard files. + +{{< admonition type="note" >}} +For production system, we recommend using the `folderFromFilesStructure` capability instead of **Administration** > **Provisioning** to include dashboards from a local file system in your Grafana instance. +Refer to [Provision Grafana](https://grafana.com/docs/grafana//administration/provisioning/#provision-folders-structure-from-filesystem-to-grafana) for more information. +{{< /admonition >}} + +### Limitations + +- A provisioned dashboard can't be deleted from within Grafana UI. The dashboard has to be deleted at the local file system and those changes synced to Grafana. +- Changes from the local file system are one way: you can't save changes from + +## Before you begin + +To set up file provisioning, you need: + +- Administration rights in your Grafana organization. +- A local directory where your dashboards will be stored. + - If you want to use a GitHub repository, refer to [Set up Git Sync](https://grafana.com/docs/grafana//observability-as-code/provision-resources/file-path-setup/). +- To update the `permitted_provisioning_paths` section of `custom.ini`. +- To enable the required feature toggles in your Grafana instance. + +## Enable required feature toggles and configure permitted paths + +To activate local file provisioning in Grafana, you need to enable the `provisioning` and `kubernetesDashboards` feature toggles. +For additional information about feature toggles, refer to [Configure feature toggles](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/feature-toggles). + +The local setting must be a relative path and its relative path must be configured in the `permitted_provisioned_paths` configuration option. +The configuration option is relative to your working directory, i.e. where you are running Grafana from; this is usually `/usr/share/grafana` or similar. + +Local file paths can point to any directory that is permitted by the configuration. +The default paths is `devenv/dev-dashboards` and `conf/provisioning` in your `grafana` installation directory. + +The path must behave as a standard file directory on the system of choice. +Any subdirectories are automatically included. + +The values that you enter for the `permitted_provisioning_paths` become the base paths for those entered when you enter a local path in the **Connect to local storage** wizard. + +1. Open your Grafana configuration file, either `grafana.ini` or `custom.ini`. For file location based on operating system, refer to [Configuration file location](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/feature-toggles/#experimental-feature-toggles). +1. Locate or add a `[feature_toggles]` section. Add these values: + + ```ini + [feature_toggles] + provisioning = true + kubernetesDashboards = true ; use k8s from browser + + # If you want easy kubectl setup development mode + grafanaAPIServerEnsureKubectlAccess = true + ``` + +1. Locate or add a `[paths]` section. To add more than one location, use the pipe character (`|`) to separate the paths. The list should not include empty paths or trailing pipes. Add these values: + + ```ini + [paths] + ; This is devenv/dev-dashboards and conf/provisioning by default. + permitted_provisioning_paths = grafana/ | /etc/grafana/provisioning/ + ``` + +1. Save the changes to the file and start Grafana. + +## Set up file-based provisioning + +To use file-based provisioning, you need the file path to the `grafana` folder where your dashboards are stored in the repository. + +To start setting up file-based provisioning: + +1. Log in to your Grafana server with an account that has the Grafana Admin flag set. +1. Select **Administration** in the left-side menu and then **Provisioning**. +1. Select [Configure file provisioning](#set-up-file-based-provisioning). + +### Connect to local storage + +The local path can point to any directory that is permitted by the configuration. +Refer to [Enabled required feature toggles and paths](#enable-required-feature-toggles-and-configure-permitted-paths) for information. + +The starting path is always your working `grafana` directory. +The prefix that must be entered is determined by the locations configured in `permitted_provisioning_paths`. +The default paths are `devenv/dev-dashboards` and `conf/provisioning` in your `grafana` installation directory. +The value you enter in the Grafana UI must _begin_ with any of the configured values. For example, `conf/provisioning/test` is valid, but `conf/test` is not. + +1. Enter the **Local path**, for example `grafana/`. This must begin with any of the configured `permitted_provisioned_paths`. +1. Select **Choose what to synchronize**. + +The set up process verifies the path and provides an error message if a problem occurs. + +### Choose what to synchronize + +In this section, you determine the actions taken with the storage you selected. + +1. Select how resources should be handled in Grafana. + +- Choose **Sync all resources with external storage** if you want to sync and manage your entire Grafana instance through external storage. You can only have one provisioned connection with this selection. +- Choose **Sync external storage to new Grafana folder** to sync external resources into a new folder without affecting the rest of your instance. You can repeat this process for up to 10 folders. - Enter a **Display name** for the repository connection. Resources stored in this connection appear under the chosen display name in the Grafana UI. + + +1. Select **Synchronize** to continue. + +### Synchronize with external storage + +After this one time step, all future updates are automatically saved to the local file path and provisioned back to the instance. + +During the initial synchronization, your dashboards will be temporarily unavailable. No data or configurations will be lost. +How long the process takes depends upon the number of resources involved. + +Select **Begin synchronization** to start the process. + +### Choose additional settings + +If you wish, you can make any files synchronized as as **Read only** so no changes can be made to the resources through Grafana. +Any resources made outside of Grafana and saved to the local repository will be reflected in Grafana. + +Select **Finish**. + +## Verify your dashboards in Grafana + +To verify that your dashboards are available at the location that you specified, click **Dashboards**. The name of the dashboard is listed in the **Name** column. diff --git a/docs/sources/observability-as-code/provision-resources/git-sync-setup.md b/docs/sources/observability-as-code/provision-resources/git-sync-setup.md new file mode 100644 index 00000000000..31ae8a951a1 --- /dev/null +++ b/docs/sources/observability-as-code/provision-resources/git-sync-setup.md @@ -0,0 +1,238 @@ +--- +description: Instructions for setting up Git Sync, so you can provision GitHub repositories for use with Grafana. +keywords: + - set up + - git integration + - git sync + - github +labels: + products: + - enterprise + - oss +title: Set up Git Sync +weight: 100 +--- + +# Set up Git Sync + +{{< admonition type="note" >}} +Git Sync is an [experimental feature](https://grafana.com/docs/release-life-cycle/) introduced in Grafana v12 for open source and Enterprise editions. Engineering and on-call support is not available. Documentation is either limited or not provided outside of code comments. No SLA is provided. Enable the `provisioning` and `kubernetesDashboards` feature toggles in Grafana to use this feature. Git Sync isn't available in Grafana Cloud. +{{< /admonition >}} + +- [Provision resources and sync dashboards](/docs/grafana//observability-as-code/provision-resources/) + - [Git Sync](/docs/grafana//observability-as-code/provision-resources/intro-git-sync/) + - [Set up Git Sync](/docs/grafana//observability-as-code/provision-resources/git-sync-setup/) + - [Set up file provisioning](/docs/grafana//observability-as-code/provision-resources/file-path-setup/) + - [Work with provisioned dashboards](/docs/grafana//observability-as-code/provision-resources/provisioned-dashboards/) + - [Manage provisioned repositories with Git Sync](/docs/grafana/observability-as-code/provision-resources/use-git-sync/) + +
+ +Git Sync lets you manage Grafana dashboards as code by storing dashboards JSON files and folders in a remote GitHub repository. +Alternatively, you can configure a local file system instead of using GitHub. +Refer to [Set up file provisioning](https://grafana.com/docs/grafana//observability-as-code/provision-resources/file-path-setup/) for information. + +This page explains how to use Git Sync with a GitHub repository. + +To set up Git Sync, you need to: + +1. Enable feature toggles in Grafana (first time set up). +1. Configure a connection to your GitHub repository. +1. Choose what content to sync with Grafana. +1. Optional: Extend Git Sync by enabling pull request notifications and image previews of dashboard changes. + +| Capability | Benefit | Requires | +| ----------------------------------------------------- | ------------------------------------------------------------------------------- | --------------------------------------------- | +| Adds a table summarizing changes to your pull request | Provides a convenient way to save changes back to GitHub. | Webhooks configured | +| Add a dashboard preview image to a PR | View a snapshot of dashboard changes to a pull request without opening Grafana. | Image renderer plugin and webhooks configured | + +## Performance impacts of enabling Git Sync + +Git Sync is an experimental feature and is under continuous development. + +We recommend evaluating the performance impact, if any, in a non-production environment. + +When Git Sync is enabled, the database load might increase, especially for instances with a lot of folders and nested folders. +Reporting any issues you encounter can help us improve Git Sync. + +## Before you begin + +To set up Git Sync, you need: + +- Administration rights in your Grafana organization. +- Enable the required feature toggles in your Grafana instance. Refer to [Enable required feature toggles](#enable-required-feature-toggles) for instructions. +- A GitHub repository to store your dashboards in. + - If you want to use a local file path, refer to [the local file path guide](https://grafana.com/docs/grafana//observability-as-code/provision-resources/file-path-setup/). +- A GitHub access token. The Grafana UI will also explain this to you as you set it up. +- Optional: A public Grafana instance. +- Optional: Image Renderer plugin to save image previews with your PRs. + +## Enable required feature toggles + +To activate Git Sync in Grafana, you need to enable the `provisioning` and `kubernetesDashboards` feature toggles. +For additional information about feature toggles, refer to [Configure feature toggles](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/feature-toggles). + +To enable the required feature toggles, add them to your Grafana configuration file: + +1. Open your Grafana configuration file, either `grafana.ini` or `custom.ini`. For file location based on operating system, refer to [Configuration file location](https://grafana.com/docs/grafana//setup-grafana/configure-grafana/feature-toggles/#experimental-feature-toggles). +1. Locate or add a `[feature_toggles]` section. Add these values: + + ```ini + [feature_toggles] + provisioning = true + kubernetesDashboards = true ; use k8s from browser + + # If you want easy kubectl setup development mode + grafanaAPIServerEnsureKubectlAccess = true + ``` + +1. Save the changes to the file and restart Grafana. + +## Create a GitHub access token + +Whenever you connect to a GitHub repository, you need to create a GitHub access token with specific repository permissions. +This token needs to be added to your Git Sync configuration to enable read and write permissions between Grafana and GitHub repository. + +1. Create a new token using [Create new fine-grained personal access token](https://github.com/settings/personal-access-tokens/new). Refer to [Managing your personal access tokens](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) for instructions. +1. Under **Permissions**, expand **Repository permissions**. +1. Set these permissions for Git Sync: + + - **Contents**: Read and write permission + - **Metadata**: Read-only permission + - **Pull requests**: Read and write permission + - **Webhooks**: Read and write permission + +1. Select any additional options and then press **Generate token**. +1. Verify the options and select **Generate token**. +1. Copy the access token. Leave the browser window available with the token until you've completed configuration. + +GitHub Apps are not currently supported. + +## Set up the connection to GitHub + +Use **Provisioning** to guide you through setting up Git Sync to use a GitHub repository. + +1. Log in to your Grafana server with an account that has the Grafana Admin flag set. +1. Select **Administration** in the left-side menu and then **Provisioning**. +1. Select **Configure Git Sync**. + +### Connect to external storage + +To connect your GitHub repository, follow these steps: + +1. Paste your GitHub personal access token into **Enter your access token**. Refer to [Create a GitHub access token](#create-a-github-access-token) for instructions. +1. Paste the **Repository URL** for your GitHub repository into the text box. +1. Enter a branch to use. The default value is `main`. +1. Add a **Path** to a subdirectory where your dashboards are stored. The default value is `grafana/`. If your dashboards are stored in the root of your repository, then remove the directory name. +1. Select **Choose what to synchronize** to have the connection to your repository verified and continue setup. + +### Choose what to synchronize + +You can choose to either use one repository for an entire organization or to a new Grafana folder (up to 10 connections). +If you choose to sync all resources with external storage, then all of your dashboards are synced to that one repository. +You won't have the option of setting up additional repositories to connect to. + +You can choose to synchronize all resources with GitHub or you can sync resources to a new Grafana folder. +The options you have depend on the status of your GitHub repository. +For example, if you are syncing with a new or empty repository, you won't have an option to migrate dashboards. + +1. Select how resources should be handled in Grafana. + +- Choose **Sync all resources with external storage** if you want to sync and manage your entire Grafana instance through external storage. You can only have one provisioned connection with this selection. +- Choose **Sync external storage to new Grafana folder** to sync external resources into a new folder without affecting the rest of your instance. You can repeat this process for up to 10 connections. - Enter a **Display name** for the repository connection. Resources stored in this connection appear under the chosen display name in the Grafana UI. + + +1. Select **Synchronize** to continue. + + + + +### Choose additional settings + +Finally, you can set up how often your configured storage is polled for updates. + +1. For **Update instance interval (seconds)**, enter how often you want the instance to pull updates from GitHub. The default value is 60 seconds. +1. Optional: Select **Read only** to ensure resources can't be modified in Grafana. + +1. Optional: If you have the Grafana Image Renderer plugin configured, you can **Enable dashboards previews in pull requests**. If image rendering is not available, then you can't select this option. For more information, refer to [Grafana Image Renderer](https://grafana.com/grafana/plugins/grafana-image-renderer/). +1. Select **Finish** to proceed. + +## Verify your dashboards in Grafana + +To verify that your dashboards are available at the location that you specified, click **Dashboards**. The name of the dashboard is listed in the **Name** column. + +Now that your dashboards have been synced from a repository, you can customize the name, change the branch, and create a pull request (PR) for it. +Refer to [Use Git Sync](https://grafana.com/docs/grafana//observability-as-code/provision-resources/use-git-sync/) for more information. + +## Configure webhooks and image rendering + +You can extend Git Sync by getting instant updates and pull requests using webhooks and add dashboard previews in pull requests. + +### Set up webhooks for realtime notification and pull request integration + +When connecting to a GitHub repository, Git Sync use webhooks to enable real-time updates from GitHub public repositories or enable the pull request integration. +Without webhooks, the polling interval is set in the final configuration screen (default is 60 seconds). +Your Grafana instance must be exposed to the public internet. +You can do this via port forwarding and DNS, a tool such as `ngrok`, or any other method you prefer. + +The permissions set in your GitHub access token provide the authorization for this communication. + +If you use local storage, then Git Sync only provides periodic pulling. + + + +Set up webhooks with whichever service or tooling you prefer. +For example, you can use Cloudflare Tunnels with a Cloudflare-managed domain, port-forwarding and DNS options, or a tool such as `ngrok`. + +After you have the public URL, you can add it to your Grafana configuration file: + +```yaml +[server] +root_url = https://PUBLIC_DOMAIN.HERE +``` + +You can check the configured webhooks in the **View** link for your GitHub repository from **Administration** > **Provisioning**. + +#### Necessary paths + +If your security setup does not permit publicly exposing the Grafana instance, you can either choose to allowlist the GitHub IP addresses, or expose only the necessary paths. + +The necessary paths required to be exposed are (RegExp): + +- `/apis/provisioning\.grafana\.app/v0(alpha1)?/namespaces/[^/]+/repositories/[^/]+/(webhook|render/.*)$` + + +### Set up image rendering for dashboard previews + +By setting up image rendering, you can add visual previews of dashboard updates directly in pull requests. +Image rendering also requires webhooks. + +You can enable this capability by installing the Grafana Image Renderer plugin in your Grafana instance. +For more information and installation instructions, refer to [Grafana Image Renderer](https://grafana.com/grafana/plugins/grafana-image-renderer/). + +## Modify configurations after set up is complete + +To update your repository configuration after you've completed set up: + +1. Log in to your Grafana server with an account that has the Grafana Admin flag set. +1. Select **Administration** in the left-side menu and then **Provisioning**. +1. Select **Settings** for the repository you wish to modify. +1. Use the **Configure repository** screen to update any of the settings. +1. Select **Save** to preserve the updates. diff --git a/docs/sources/observability-as-code/provision-resources/intro-git-sync.md b/docs/sources/observability-as-code/provision-resources/intro-git-sync.md new file mode 100644 index 00000000000..59dbf53695d --- /dev/null +++ b/docs/sources/observability-as-code/provision-resources/intro-git-sync.md @@ -0,0 +1,88 @@ +--- +description: Learn about Git Sync, the Grafana feature for storing and managing dashboards within GitHub repositories. +keywords: + - dashboards + - git integration + - git sync + - github +labels: + products: + - enterprise + - oss +title: Git Sync +weight: 100 +--- + +# Git Sync + +{{< admonition type="caution" >}} +Git Sync is an [experimental feature](https://grafana.com/docs/release-life-cycle/) introduced in Grafana v12 for open source and Enterprise editions. Engineering and on-call support is not available. Documentation is either limited or not provided outside of code comments. No SLA is provided. Enable the `provisioning` and `kubernetesDashboards` feature toggles in Grafana to use this feature. Git Sync isn't available in Grafana Cloud. +{{< /admonition >}} + +- [Provision resources and sync dashboards](/docs/grafana//observability-as-code/provision-resources/) + - [Git Sync](/docs/grafana//observability-as-code/provision-resources/intro-git-sync/) + - [Set up Git Sync](/docs/grafana//observability-as-code/provision-resources/git-sync-setup/) + - [Set up file provisioning](/docs/grafana//observability-as-code/provision-resources/file-path-setup/) + - [Work with provisioned dashboards](/docs/grafana//observability-as-code/provision-resources/provisioned-dashboards/) + - [Manage provisioned repositories with Git Sync](/docs/grafana/observability-as-code/provision-resources/use-git-sync/) + +
+ +Using Git Sync, you can: + +- Introduce a review process for creating and modifying dashboards +- Manage dashboard configuration outside of Grafana instances +- Replicate dashboards across multiple instances + +Whenever a dashboard is modified, Grafana can commit changes to Git upon saving. Users can configure settings to either enforce PR approvals before merging or allow direct commits. + +Users can push changes directly to GitHub and see them in Grafana. Similarly, automated workflows can do changes that will be automatically represented in Grafana by updating Git. + +Because the dashboards are defined in JSON files, you can enable as-code workflows where the JSON is output from Go, TypeScript, or another coding language in the format of a dashboard schema. + +To learn more about creating dashboards in a coding language to provision them for Git Sync, refer to the [Foundation SDK](https://grafana.com/docs/grafana//observability-as-code/foundation-sdk) documentation. + +## How it works + +Git Sync is bidirectional and also works with changes done directly in GitHub as well as within the Grafana UI. +Grafana periodically polls GitHub at a regular internal to synchronize any changes. +With the webhooks feature enabled, repository notifications appear almost immediately. +Without webhooks, Grafana polls for changes at the specified interval. +The default polling interval is 60 seconds. + +Any changes made in the provisioned files stored in the GitHub repository are reflected in the Grafana database. +The Grafana UI reads the database and updates the UI to reflect these changes. + +## Common use cases + +Git Sync in Grafana lets you manage dashboards as code. +Because your dashboard JSON files are stored in GitHub, you and your team can version control, collaborate, and automate deployments efficiently. + +### Version control and auditing + +Organizations can maintain a structured, version-controlled history of Grafana dashboards. +The version control lets you revert to previous versions when necessary, compare modifications across commits, and ensure transparency in dashboard management. +Additionally, having a detailed history of changes enhances compliance efforts, as teams can generate audit logs that document who made changes, when they were made, and why. + +### Automated deployment and CI/CD integration + +Teams can streamline their workflow by integrating dashboard updates into their CI/CD pipelines. +By pushing changes to GitHub, automated processes can trigger validation checks, test dashboard configurations, and deploy updates programmatically using the `grafanactl` CLI and Foundation SDK. +This reduces the risk of human errors, ensures consistency across environments, and enables a faster, more reliable release cycle for dashboards used in production monitoring and analytics. + +### Collaborative dashboard development + +With Git Sync, multiple users can work on dashboards simultaneously without overwriting each other’s modifications. +By leveraging pull requests and branch-based workflows, teams can submit changes for review before merging them into the main branch. This process not only improves quality control but also ensures that dashboards adhere to best practices and organizational standards. Additionally, GitHub’s built-in discussion and review tools facilitate effective collaboration, making it easier to address feedback before changes go live. + +### Multi-environment synchronization + +Enterprises managing multiple Grafana instances, such as development, staging, and production environments, can seamlessly sync dashboards across these instances. +This ensures consistency in visualization and monitoring configurations, reducing discrepancies that might arise from manually managing dashboards in different environments. +By using Git Sync, teams can automate deployments across environments, eliminating repetitive setup tasks and maintaining a standardized monitoring infrastructure across the organization. + +### Disaster recovery and backup + +By continuously syncing dashboards to GitHub, organizations can create an always-updated backup, ensuring dashboards are never lost due to accidental deletion or system failures. +If an issue arises--such as a corrupted dashboard, unintended modification, or a system crash--teams can quickly restore the latest functional version from the Git repository. +This not only minimizes downtime but also adds a layer of resilience to Grafana monitoring setups, ensuring critical dashboards remain available when needed. diff --git a/docs/sources/observability-as-code/provision-resources/provisioned-dashboards.md b/docs/sources/observability-as-code/provision-resources/provisioned-dashboards.md new file mode 100644 index 00000000000..4b03d96ee61 --- /dev/null +++ b/docs/sources/observability-as-code/provision-resources/provisioned-dashboards.md @@ -0,0 +1,137 @@ +--- +description: Update, save, and modify provisioned resources in Grafana using Git Sync. +keywords: + - dashboards + - provisioned files + - git sync + - github +labels: + products: + - enterprise + - oss +title: Work with provisioned dashboards +weight: 300 +--- + +# Work with provisioned dashboards + +{{< admonition type="note" >}} +Git Sync and File path provisioning an [experimental feature](https://grafana.com/docs/release-life-cycle/) introduced in Grafana v12 for open source and Enterprise editions. Engineering and on-call support is not available. Documentation is either limited or not provided outside of code comments. No SLA is provided. Enable the `provisioning` and `kubernetesDashboards` feature toggles in Grafana. These features aren't available in Grafana Cloud. +{{< /admonition >}} + +- [Provision resources and sync dashboards](/docs/grafana//observability-as-code/provision-resources/) + - [Git Sync](/docs/grafana//observability-as-code/provision-resources/intro-git-sync/) + - [Set up Git Sync](/docs/grafana//observability-as-code/provision-resources/git-sync-setup/) + - [Set up file provisioning](/docs/grafana//observability-as-code/provision-resources/file-path-setup/) + - [Work with provisioned dashboards](/docs/grafana//observability-as-code/provision-resources/provisioned-dashboards/) + - [Manage provisioned repositories with Git Sync](/docs/grafana/observability-as-code/provision-resources/use-git-sync/) + +
+ +Using Provisioning, you can choose to store your dashboard JSON files in either GitHub repositories using Git Sync or a local file path. + +For more information, refer to the [Dashboards](https://grafana.com/docs/grafana//dashboards/) documentation. + +## Provisioning methods + +Dashboards and folders synchronized using Git Sync or a local file path are referred to as "provisioned" resources. + +Of the two experimental options, Git Sync is the recommended method for provisioning your dashboards. +You can synchronize any new dashboards and changes to existing dashboards to your configured GitHub repository. +If you push a change in the repository, those changes are mirrored in your Grafana instance. +For more information on configuring Git Sync, refer to [Set up Git Sync](https://grafana.com/docs/grafana//observability-as-code/provision-resources/intro-git-sync/). + +### Local path provisioning + +Using the local path provisioning makes files from a specified path available within Grafana. +These provisioned resources can only be modified in the local files and not within Grafana. +Any changes made in the configured local path are updated in Grafana. + +Refer to [Set up file provisioning](https://grafana.com/docs/grafana//observability-as-code/provision-resources/file-path-setup) to learn more about the version of local file provisioning in Grafana 12. + +{{< admonition type="note" >}} +The experimental local path provisioning using **Administration** > **Provisioning** will replace the file provisioning methods Grafana uses for referencing local file. + +For production systems, use the established methods for provisioning file systems in Grafana. +Refer to [Provision Grafana](https://grafana.com/docs/grafana//administration/provisioning/#provision-folders-structure-from-filesystem-to-grafana) for more information. +{{< /admonition >}} + +## Manage dashboards provisioned with Git Sync + +Using Git Sync, you can manage your dashboards in the UI and synchronize them with a GitHub repository. + +Git Sync changes the behavior in Grafana for dashboards that are saved in Git Sync: + +- Dashboards saved in your repository or local folder configured with Git Sync appear in a provisioned folder in Grafana. +- Any dashboard folders saved with Git Sync have a **Provisioned** label in the UI. +- Any changes to a provisioned resources have to be saved to the repository by opening a pull request or committing directly to the `main` branch. + +You can set a single folder, or multiple folders to a different repository, with up to 10 connections. + +### Git workflow with dashboards + +By default, Git version control uses a branch-based workflow for changes. This means that you can: + +- Commit changes to an existing branch (such as `main`) or save them to a new branch in your GitHub repository. +- Use pull requests to review changes to dashboards. +- Preview the changes before merging. + +To learn more about Git, refer to [Getting Started - About Version Control](https://git-scm.com/book/en/v2/Getting-Started-About-Version-Control) of the [Pro Git book](https://git-scm.com/book/en/v2) in the official Git documentation. + +### Add and save a new dashboard + +When you create a new dashboard in a provisioned folder associated with a GitHub repository, you follow the same process you use for any new dashboard. +Refer to [Create a dashboard](http://grafana.com/docs/grafana//dashboards/build-dashboards/create-dashboard/) for more information. + +After you create the dashboard, the steps are similar to [Save dashboard changes to GitHub](#save-dashboard-changes-to-github). + +1. Select **Save** to preserve the new dashboard. +1. Enter a title for the dashboard and a description. +1. Select the provisioned folder from the **Folder** drop-down list. +1. In **Path**, provide the path for your repository, ending in a JSON or YAML file. +1. For **Workflow**, select **Push to main** to make a Git commit directly to the repository or **Push to a new branch** to create a pull request. + - **Branch**: Specify the branch name in GitHub (for example, main). This option only appears if you select **Push to a new branch**. +1. Select **Save**. + +### Save dashboard changes to GitHub + +When you edit a provisioned resource, you are prompted to save or discard those changes. +Saving changes requires opening a pull request in your GitHub repository. + +1. Select **Edit** to update a provisioned dashboard. Make your desired changes. + +1. Click **Save dashboard**. + +1. On the **Provisioned dashboard** panel, choose the options you want to use: + + - **Update default refresh value**: Check this box to make the current refresh the new default. + - **Update default variable values**: Check this box to make the current values the new default. + - **Path**: Provide the path for your repository, ending in a JSON or YAML file. + - **Workflow:** Select **Push to main** to make a Git commit directly to the repository or **Push to a new branch** to create a pull request. + - **Branch**: Specify the branch name in GitHub (for example, main). This option only appears if you select **Push to a new branch**. + - **Comment**: Add a comment describing your changes. + +1. Optional: Select the **Changes** tab to view the differences between the updates you made and the original resource. + +1. Select **Save**. + +1. If you chose **Push to a new branch**, select **Open a pull request in GitHub** to open a new PR to your repository. GitHub opens with your dashboard’s code as the contents of the PR. + +1. Follow your usual GitHub workflow to save and merge the PR to your repository. + +### Tips + +- Use GitHub pull requests for changes to maintain review processes. +- Provide clear commit messages describing your changes. +- Regularly sync your repository to keep Grafana up to date. +- Review the **Events** tab to monitor sync status. + +## Manage dashboards provisioned with file provisioning + +To update any resources in the local path, you need to edit the files directly and then save them locally. +These changes are synchronized to Grafana. +However, you can't create, edit, or delete these resources using the Grafana UI. + +For more information, refer to [How it works](https://grafana.com/docs/grafana//observability-as-code/provision-resources/). + +Refer to [Set up file provisioning](https://grafana.com/docs/grafana//observability-as-code/provision-resources/file-path-setup/) for configuration instructions. diff --git a/docs/sources/observability-as-code/provision-resources/use-git-sync.md b/docs/sources/observability-as-code/provision-resources/use-git-sync.md new file mode 100644 index 00000000000..192a73226e1 --- /dev/null +++ b/docs/sources/observability-as-code/provision-resources/use-git-sync.md @@ -0,0 +1,84 @@ +--- +description: Instructions for working with Git Sync to perform common tasks, such as saving dashboards to GitHub and synchronizing changes with Grafana. +keywords: + - as code + - as-code + - dashboards + - git integration + - git sync + - github +labels: + products: + - enterprise + - oss +title: Manage provisioned repositories with Git Sync +menuTitle: Manage repositories +weight: 400 +--- + +# Manage provisioned repositories with Git Sync + +- [Provision resources and sync dashboards](/docs/grafana//observability-as-code/provision-resources/) + - [Git Sync](/docs/grafana//observability-as-code/provision-resources/intro-git-sync/) + - [Set up Git Sync](/docs/grafana//observability-as-code/provision-resources/git-sync-setup/) + - [Set up file provisioning](/docs/grafana//observability-as-code/provision-resources/file-path-setup/) + - [Work with provisioned dashboards](/docs/grafana//observability-as-code/provision-resources/provisioned-dashboards/) + - [Manage provisioned repositories with Git Sync](/docs/grafana/observability-as-code/provision-resources/use-git-sync/) + +
+ +After you have set up Git Sync, you can synchronize dashboards and changes to existing dashboards to your configured GitHub repository. +If you push a change in the repository, those changes are mirrored in your Grafana instance. + +## View current status of synchronization + +Each repository synchronized with Git Sync has a dashboard that provides a summary of resources, health, pull status, webhook, sync jobs, resources, and files. +Use the detailed information accessed in **View** to help troubleshoot and understand the health of your repository's connection with Grafana. + +To view the current status, follow these steps. + +1. Log in to your Grafana server with an account that has the Grafana Admin or Editor flag set. +1. Select **Administration** in the left-side menu and then **Provisioning**. +1. Locate the repository you are interested in. +1. If you see a green `Up-to-date` label next to the repository name, then everything is syncing as expected. +1. Select **View** to access detailed dashboards and reports about the synchronization history of your repository. + +## Synchronize changes + +Synchronizing resources from provisioned repositories into your Grafana instance pulls the resources into the selected folder. Existing dashboards with the same `uid` are overwritten. + +To sync changes from your dashboards with your Git repository: + +1. From the left menu, select **Administration** > **Provisioning**. +1. Select **Pull** under the repository you want to sync. +1. Wait for the synchronization process to complete. + +## Remove a repository + +To delete a repository, follow these steps. + +1. Log in to your Grafana server with an account that has the Grafana Admin or Editor flag set. +1. Select **Administration** in the left-side menu and then **Provisioning**. +1. Locate the repository you are interested in. +1. Select the trashcan icon in the right side to delete the chosen entry. +1. Select **Delete** to confirm. + +## Troubleshoot synchronization + +Monitor the **View** status page for synchronization issues and status updates. Common events include: + +- Sync started +- Sync completed +- Sync failed (with error details) +- Sync issues + +### Dashboard sync errors + +- If dashboards are not syncing, check if the repository URL is correct and accessible from the Grafana instance. +- Ensure that the configured repository branch exists and is correctly referenced. +- Check for conflicts in the repository that may prevent syncing. + +### Dashboard import errors + +- Validate the JSON format of the dashboard files before importing. +- If the import fails, check Grafana logs for error messages and troubleshoot accordingly.