public/grafana
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Provisioning: Fix race condition causing unhealthy repository message to be lost (#115150)

Browse Source 
* Fix race condition causing unhealthy repository message to be lost

This commit fixes a race condition in the provisioning repository controller
where the "Repository is unhealthy" message in the sync status could be lost
due to status updates being based on stale repository objects.

## Problem

The issue occurred in the `process` function when:
1. Repository object was fetched from cache with old status
2. `RefreshHealth` immediately patched the health status to "unhealthy"
3. `determineSyncStatusOps` used the stale object to check if unhealthy
   message was already set
4. A second patch operation based on stale data would overwrite the
   health status update

## Solution

Introduced `RefreshHealthWithPatchOps` method that returns patch operations
instead of immediately applying them. This allows batching all status updates
(health + sync) into a single atomic patch operation, eliminating the race
condition.

## Changes

- Added `HealthCheckerInterface` for better testability
- Added `RefreshHealthWithPatchOps` method to return patch ops without applying
- Updated `process` function to batch health and sync status updates
- Added comprehensive unit tests for the fix

Fixes the issue where unhealthy repositories don't show the "Repository is
unhealthy" message in their sync status.

* Fix staticcheck lint error: remove unnecessary nil check for slice
This commit is contained in:
Roberto Jiménez Sánchez
2025-12-12 13:24:58 +02:00
committed by GitHub
parent c7c052480d
commit b863acab05
6 changed files with 518 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pkg/registry/apis/provisioning/controller/repository.go
+6 -1
View File
@@ -561,11 +561,16 @@ func (rc *RepositoryController) process(item *queueItem) error {
}
// Handle health checks using the health checker
_, healthStatus, err := rc.healthChecker.RefreshHealth(ctx, repo)
_, healthStatus, healthPatchOps, err := rc.healthChecker.RefreshHealthWithPatchOps(ctx, repo)
if err != nil {
return fmt.Errorf("update health status: %w", err)
}
// Add health patch operations first
if len(healthPatchOps) > 0 {
patchOperations = append(patchOperations, healthPatchOps...)
}
// determine the sync strategy and sync status to apply
syncOptions := rc.determineSyncStrategy(ctx, obj, repo, shouldResync, healthStatus)
patchOperations = append(patchOperations, rc.determineSyncStatusOps(obj, syncOptions, healthStatus)...)