chore: add documentation on unified search (#108847)

* chore: add documentation on unified search Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com> * fix: add additional needed feature flags for unified search Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com> * fix: add docs about index_min_count and index_max_count Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com> * fix: add documentation about sortable fields and the discrepancy with search/sortable Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com> * fix: kubernetesClientDashboardsFolders feature flag is no more, remove it from the docs Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com> * fix: simplify request flow diagrams Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com> --------- Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>
2025-07-31 07:18:29 +00:00
parent ef80129bd9
commit 91e5e8019d
1 changed files with 509 additions and 0 deletions
@@ -777,3 +777,512 @@ The dual writer system provides metrics for monitoring:
 - `dual_writer_mode_transitions_total`: Counter of mode transitions

 Use these metrics to monitor the health of your migration and identify any issues with the dual writer system.
+
+---
+
+## Unified Search System
+
+The Unified Search system provides a scalable, distributed search capability for Grafana's Unified Storage. It uses a ring-based architecture to distribute search requests across multiple search server instances, with namespace-based sharding for optimal performance and data distribution.
+
+### System Architecture
+
+The search system provides both unified and legacy search capabilities, with routing based on dual writer mode configuration:
+
+```mermaid
+graph TB
+    subgraph "Request Sources"
+        A[Grafana UI<br/>User Search]
+        B[Dashboard Service<br/>Resource Searches]
+        C[Folder Service<br/>Resource Searches]
+        D[Alerting Service<br/>Resource Searches]
+        E[Provisioning Service<br/>Resource Searches]
+        F[API Endpoints<br/>Search Operations]
+    end
+    
+    subgraph "API Gateway Layer"
+        G[Grafana API Server<br/>Search Endpoint]
+        H[Search Client<br/>Dual Writer Aware]
+    end
+    
+    subgraph "Routing Decision"
+        I{Dual Writer Mode<br/>Check}
+    end
+    
+    subgraph "Unified Search Path (Mode 3+)"
+        J[Search Distributor<br/>Ring-based Routing]
+        K[Ring<br/>Consistent Hashing]
+        L[Search API Server 1<br/>Namespace Sharding<br/>+ Embedded Bleve Backend]
+        M[Search API Server 2<br/>Namespace Sharding<br/>+ Embedded Bleve Backend]
+        N[Search API Server 3<br/>Namespace Sharding<br/>+ Embedded Bleve Backend]
+        O[Unified Storage<br/>K8s-style Resources]
+    end
+    
+    subgraph "Legacy Search Path (Mode 0-2)"
+        Q[Legacy Search Service<br/>SQL-based Search]
+        R[Legacy Storage<br/>Traditional Tables]
+        S[Shadow Traffic<br/>Mode 1-2 + Flag Enabled]
+    end
+    
+    A --> G
+    B --> H
+    C --> H
+    D --> H
+    E --> H
+    F --> G
+    G --> H
+    H --> I
+    
+    I -->|Mode 3+| J
+    I -->|Mode 0-2| Q
+    
+    J --> K
+    K --> L
+    K --> M
+    K --> N
+    L -.->|Just-in-Time<br/>Indexing| O
+    M -.->|Just-in-Time<br/>Indexing| O
+    N -.->|Just-in-Time<br/>Indexing| O
+    
+    Q --> R
+    Q -.->|Shadow Traffic<br/>Mode 1-2 + Flag| S
+    S -.-> J
+```
+
+### Search Backend Routing
+
+The search client routes requests based on the dual writer mode configuration for each resource type:
+
+#### Dual Writer Mode → Backend Routing
+- **Mode 0-2**: Route to **Legacy Search**
+  - Mode 1-2: Shadow traffic to Unified Search (if `unifiedStorageSearchDualReaderEnabled` is enabled)
+  - Mode 0: No shadow traffic
+- **Mode 3+**: Route to **Unified Search** 
+  - No shadow traffic needed (unified is primary)
+
+### Feature Flags
+
+Unified Search requires several feature flags to be enabled depending on the desired functionality:
+
+#### Prerequisites (Required for Unified Storage)
+
+| Feature Flag | Purpose | Stage | Required For |
+|--------------|---------|-------|--------------|
+| `grafanaAPIServerWithExperimentalAPIs` | Allow experimental API groups | Development | Access to v0alpha1 APIs (including search) |
+
+#### Unified Search Specific Flags
+
+| Feature Flag | Purpose | Stage | Required For |
+|--------------|---------|-------|--------------|
+| `unifiedStorageSearch` | Core search functionality | Experimental | Search API servers, indexing |
+| `unifiedStorageSearchUI` | Frontend search interface | Experimental | Grafana UI search |
+| `unifiedStorageSearchPermissionFiltering` | User permission filtering | GA | Access control in search results |
+| `unifiedStorageSearchSprinkles` | Usage insights integration | Experimental | Dashboard usage sorting (Enterprise) |
+| `unifiedStorageSearchDualReaderEnabled` | Shadow traffic to unified search | Experimental | Shadow traffic during migration |
+
+#### Basic Configuration
+```ini
+[feature_toggles]
+; Prerequisites for unified storage (required)
+grafanaAPIServerWithExperimentalAPIs = true
+
+; Core search functionality (required)
+unifiedStorageSearch = true
+
+; Enable search UI (required for frontend)
+unifiedStorageSearchUI = true
+
+; Enable permission filtering (recommended)
+unifiedStorageSearchPermissionFiltering = true
+
+; Enable shadow traffic during migration (optional)
+unifiedStorageSearchDualReaderEnabled = true
+
+; Enable usage insights sorting (Enterprise only)
+unifiedStorageSearchSprinkles = true
+```
+
+### Request Flow Diagrams
+
+#### Search Request Flow with Dual Writer Mode Routing
+
+Search requests originate from multiple sources, and the search client routes based on dual writer mode configuration:
+
+```mermaid
+flowchart TD
+    A[Search Request] --> B{Dual Writer Mode}
+    B -->|Mode 3+| C[Unified Search]
+    B -->|Mode 0-2| D[Legacy Search]
+    C --> E[Return Results]
+    D --> E
+```
+
+#### Search Request Flow with Shadow Traffic
+
+When `unifiedStorageSearchDualReaderEnabled` is enabled and resource is in dual writer Mode 1-2 (legacy primary), shadow traffic is generated:
+
+```mermaid
+flowchart TD
+    A[Search Request] --> B{Shadow Traffic Enabled?}
+    B -->|Yes| C[Primary: Legacy Search]
+    B -->|No| D[Single Search Path]
+    C --> E[Background: Unified Search]
+    C --> F[Return Legacy Results]
+    E --> G[Log Results for Comparison]
+    D --> H[Return Results]
+```
+
+### Distributor Architecture
+
+The Search Distributor acts as a smart proxy that routes search requests to the appropriate search API server based on namespace hashing:
+
+```mermaid
+flowchart TD
+    A[Incoming Search Request] --> B[Hash Namespace]
+    B --> C[Select Random Instance]
+    C --> D[Proxy Request]
+    D --> E[Return Response]
+```
+
+#### Key Features:
+- **Namespace-based routing**: Each request is routed based on the target namespace
+- **Load balancing**: Random selection among available replicas for the namespace
+- **Health awareness**: Only routes to `ACTIVE` ring instances
+- **Connection pooling**: Reuses gRPC connections for efficiency
+- **Proxy headers**: Adds metadata for debugging and tracing
+
+### Ring Architecture
+
+The hash ring provides consistent, distributed assignment of namespaces to search API servers:
+
+```mermaid
+flowchart TD
+    A[Namespace] --> B[Hash Function]
+    B --> C[Ring Position]
+    C --> D[Assigned Instance]
+    D --> E[Search Processing]
+```
+
+#### Ring Properties:
+- **Consistent hashing**: Uses FNV32 hash function for namespace distribution
+- **128 tokens per instance**: Provides good distribution across the ring
+- **Replication factor**: Configurable redundancy (default based on cluster size)
+- **State management**: Instances transition through JOINING → ACTIVE → LEAVING
+- **Automatic rebalancing**: Ring adjusts when instances join/leave
+
+### Namespace-Based Sharding
+
+Unified Search uses namespace-based sharding to distribute search indexes across multiple search API servers:
+
+```mermaid
+flowchart LR
+    A[Namespaces] --> B[Hash Ring]
+    B --> C[Search Server 1]
+    B --> D[Search Server 2]  
+    B --> E[Search Server 3]
+    C --> F[Indexes for Assigned Namespaces]
+    D --> G[Indexes for Assigned Namespaces]
+    E --> H[Indexes for Assigned Namespaces]
+```
+
+#### Sharding Benefits:
+1. **Horizontal scalability**: Add more search servers to handle more namespaces
+2. **Resource isolation**: Each namespace's index is independent
+3. **Parallel processing**: Multiple searches can run simultaneously across different servers
+4. **Fault tolerance**: Namespace availability depends only on its assigned server(s)
+
+#### Sharding Algorithm:
+```go
+func getSearchServer(namespace string) string {
+    hash := fnv.New32a()
+    hash.Write([]byte(namespace))
+    
+    // Get replication set from ring
+    replicationSet := ring.GetWithOptions(
+        hash.Sum32(), 
+        searchRingRead, 
+        ring.WithReplicationFactor(ring.ReplicationFactor())
+    )
+    
+    // Random load balancing within replication set
+    instance := replicationSet.Instances[rand.Intn(len(replicationSet.Instances))]
+    return instance.Id
+}
+```
+
+### Search Index Management
+
+Each search API server contains an embedded Bleve search engine that manages indexes for its assigned namespaces:
+
+```mermaid
+flowchart TD
+    A[Search Request] --> B{Index Ready?}
+    B -->|Yes| C[Query Index]
+    B -->|No| D[Build Index]
+    D --> E[Fetch Resources]
+    E --> F[Create Index]
+    F --> C
+    C --> G[Return Results]
+    
+    H[Resource Changes] --> I[Update Index]
+    I --> F
+```
+
+#### Index Architecture Details:
+
+**Embedded Bleve Backend**: Each Search API Server contains its own Bleve search engine instance, not a shared external service.
+
+**Just-in-Time Indexing**: When a search request arrives for a namespace that doesn't have an index (or has an outdated index):
+1. The Search API Server fetches all resources for that namespace from Unified Storage
+2. Builds search documents in memory
+3. Creates either a memory-based or disk-based Bleve index depending on size
+4. Executes the search query against the newly built index
+5. Returns results to the user
+
+**Index Storage Strategy**:
+- **Memory indexes**: For small datasets (< `index_file_threshold` documents)
+- **Disk indexes**: For large datasets (≥ `index_file_threshold` documents)
+- Indexes are stored per Search API Server instance, not globally shared
+
+**Background Updates**: In addition to just-in-time indexing, Search API Servers also maintain indexes through background watch events for incremental updates.
+
+#### Index Configuration:
+```ini
+[unified_storage]
+; Path for disk-based search indexes
+index_path = /var/lib/grafana/unified-search/bleve
+
+; Threshold for file-based vs memory indexes  
+index_file_threshold = 1000
+
+; Maximum batch size for indexing
+index_max_batch_size = 100
+
+; Number of worker threads for indexing
+index_workers = 4
+
+; Cache TTL for indexes
+index_cache_ttl = 1h
+
+; Periodic rebuild interval (for usage insights)
+index_rebuild_interval = 24h
+
+; Minimum resource count required to build an index (default: 1)
+; If a namespace has fewer resources than this threshold, no index will be created
+index_min_count = 1
+
+; Maximum resource count before creating an empty index (default: 0 = no limit)
+; When exceeded, creates an empty index instead of indexing all resources for performance
+index_max_count = 0
+```
+
+### Search Request Sources
+
+Unified Search serves multiple types of consumers within the Grafana ecosystem:
+
+#### 1. User-Initiated Searches
+- **Source**: Grafana UI search interface
+- **Purpose**: Interactive dashboard and folder discovery
+- **Characteristics**: Real-time, user-facing, latency-sensitive
+- **Endpoint**: `/api/v1/search` (legacy search UI) or `/apis/dashboard.grafana.app/v0alpha1/namespaces/{namespace}/search` (when `unifiedStorageSearchUI` is enabled)
+
+#### 2. Internal Service Searches
+
+Internal services use different search backends depending on dual writer mode configuration:
+
+- **Dashboard Service**: 
+  - Find related dashboards based on tags, folders, or content
+  - Discover dashboards for playlist creation
+  - Validate dashboard references during operations
+  - **Backend**: Depends on dashboard dual writer mode (Legacy for Mode 0-2, Unified for Mode 3+)
+
+- **Folder Service**:
+  - Retrieve folder contents and nested structures
+  - Resolve folder hierarchy relationships
+  - Check folder permissions and accessibility
+  - **Backend**: Depends on folder dual writer mode (Legacy for Mode 0-2, Unified for Mode 3+)
+
+- **Alerting Service**:
+  - Discover dashboards and panels for alert rule creation
+  - Find existing alert rules across namespaces
+  - Resolve dashboard/panel references in alert definitions
+  - **Backend**: Mixed - dashboard searches use dashboard dual writer mode, alert rule searches typically use legacy
+
+- **Provisioning Service**:
+  - Check for existing resources before provisioning
+  - Validate resource uniqueness and naming conflicts
+  - Discover resources for bulk operations
+  - **Backend**: Depends on each resource type's dual writer mode configuration
+
+- **API Services**:
+  - Backend support for various API endpoints
+  - Resource validation and dependency checking
+  - **Backend**: Routes based on resource type's dual writer mode
+
+#### 3. Search Operation Types
+
+Unified Search supports multiple types of search operations:
+
+##### Resource Search
+- **Purpose**: Find resources (dashboards, folders, etc.) by content
+- **Endpoint**: `/api/v1/search` (legacy) or `/apis/dashboard.grafana.app/v0alpha1/namespaces/{namespace}/search` (when `unifiedStorageSearchUI` is enabled)
+- **Additional endpoint**: `/apis/dashboard.grafana.app/v0alpha1/namespaces/{namespace}/search/sortable` for retrieving sortable fields
+- **Features**: Full-text search, filtering, sorting
+
+**Sortable Fields:**
+
+The `/search/sortable` endpoint currently returns a limited static list:
+```json
+{
+  "fields": [
+    {"field": "title", "display": "Title (A-Z)", "type": "string"},
+    {"field": "-title", "display": "Title (Z-A)", "type": "string"}
+  ]
+}
+```
+
+However, the search backend actually supports sorting by many more fields:
+
+**Standard Fields:**
+- `title` - Resource display name (uses `title_phrase` for exact sorting)
+- `name` - Kubernetes resource name
+- `description` - Resource description
+- `folder` - Parent folder name
+- `created` - Creation timestamp (int64)
+- `updated` - Last update timestamp (int64)
+- `createdBy` - Creator user ID
+- `updatedBy` - Last updater user ID
+- `tags` - Resource tags (array)
+- `rv` - Resource version (int64)
+
+**Dashboard-Specific Fields** (require `fields.` prefix):
+- `fields.schema_version` - Dashboard schema version
+- `fields.link_count` - Number of dashboard links
+- `fields.panel_types` - Panel types used in dashboard
+- `fields.ds_types` - Data source types used
+- `fields.transformation` - Transformations used
+
+**Usage Insights Fields** (Enterprise only, require `fields.` prefix):
+- `fields.views_total` - Total dashboard views
+- `fields.views_last_1_days` / `fields.views_last_7_days` / `fields.views_last_30_days` - Recent views
+- `fields.views_today` - Today's views
+- `fields.queries_total` - Total queries executed
+- `fields.queries_last_1_days` / `fields.queries_last_7_days` / `fields.queries_last_30_days` - Recent queries
+- `fields.queries_today` - Today's queries
+- `fields.errors_total` - Total errors
+- `fields.errors_last_1_days` / `fields.errors_last_7_days` / `fields.errors_last_30_days` - Recent errors
+- `fields.errors_today` - Today's errors
+
+**Usage Examples:**
+```bash
+# Sort by title (ascending)
+GET /apis/dashboard.grafana.app/v0alpha1/namespaces/{namespace}/search?sortBy=title
+
+# Sort by creation date (descending)  
+GET /apis/dashboard.grafana.app/v0alpha1/namespaces/{namespace}/search?sortBy=-created
+
+# Sort by usage insights (Enterprise)
+GET /apis/dashboard.grafana.app/v0alpha1/namespaces/{namespace}/search?sortBy=-fields.views_total
+```
+
+*Note: There's currently a discrepancy between the limited fields exposed by `/search/sortable` and the full range of fields actually supported by the search backend.*
+
+##### Federated Search  
+- **Purpose**: Search across multiple resource types simultaneously
+- **Features**: Cross-resource queries, unified result ranking, combined sorting and faceting
+- **Implementation**: Uses Bleve IndexAlias to combine multiple indexes for unified searching
+- **Default behavior**: When no type is specified, automatically federates dashboards and folders
+
+**How Federated Search Works:**
+
+Federated search is implemented using Bleve's IndexAlias feature, which allows searching across multiple indexes as if they were a single unified index. This enables:
+
+1. **Cross-resource queries**: Search for content across dashboards, folders, and other resource types
+2. **Unified sorting**: Results from different resource types are merged and sorted together
+3. **Combined faceting**: Aggregate facet statistics across all federated resource types
+4. **Permission filtering**: Respects user permissions for each resource type independently
+
+**API Usage Examples:**
+
+**1. Default Federation (Dashboards + Folders):**
+```bash
+# When no type is specified, automatically searches dashboards and folders
+GET /apis/dashboard.grafana.app/v0alpha1/namespaces/{namespace}/search?query=my-search
+```
+
+**2. Single Resource Type Search:**
+```bash
+# Search only folders (despite the "dashboard" API group, type parameter controls what's searched)
+GET /apis/dashboard.grafana.app/v0alpha1/namespaces/{namespace}/search?type=folders&query=my-search
+
+# Search only dashboards
+GET /apis/dashboard.grafana.app/v0alpha1/namespaces/{namespace}/search?type=dashboards&query=my-search
+```
+
+**3. Explicit Two-Type Federation:**
+```bash
+# Search dashboards (primary) with folders federated
+GET /apis/dashboard.grafana.app/v0alpha1/namespaces/{namespace}/search?type=dashboards&type=folders&query=my-search
+```
+
+**4. Protocol Buffer Request Structure:**
+```protobuf
+message ResourceSearchRequest {
+  ListOptions options = 1;          // Primary resource type to search
+  repeated ResourceKey federated = 2; // Additional resource types to federate
+  string query = 3;                 // Search query applied across all types
+  // ... other fields
+}
+```
+
+**Example gRPC/Protocol Buffer Usage:**
+```go
+searchRequest := &resourcepb.ResourceSearchRequest{
+    Options: &resourcepb.ListOptions{
+        Key: dashboardKey, // Primary: search dashboards
+    },
+    Federated: []*resourcepb.ResourceKey{
+        folderKey, // Also search folders
+    },
+    Query: "monitoring",
+    Limit: 50,
+    SortBy: []*resourcepb.ResourceSearchRequest_Sort{
+        {Field: "title", Desc: false}, // Sort combined results by title
+    },
+}
+```
+
+**5. Unified Results:**
+Federated search returns a single result set containing resources from all specified types, with:
+- **Unified ranking**: All results scored and ranked together
+- **Cross-type sorting**: Resources from different types sorted by common fields (title, tags, etc.)
+- **Resource type identification**: Each result includes metadata indicating its resource type
+- **Permission-aware filtering**: Only returns resources the user has permission to see
+
+**Limitations:**
+- Federation only works across resource types with **common fields** (title, tags, folder, etc.)
+- All federated indexes must be of the same search backend type (currently Bleve)
+- Currently supports up to 2 resource types in federation via the API endpoint
+- **Architectural note**: The search endpoint is under `dashboard.grafana.app` but can search any resource type via the `type` parameter - this is a design choice where the "dashboard search" has evolved into a generic search endpoint
+
+##### Managed Objects
+- **Purpose**: Administrative queries for resource management
+- **Operations**: Count, list, statistics
+
+##### Stats and Monitoring
+- **Purpose**: Index health and performance metrics
+- **Metrics**: Document counts, index sizes, search latency
+
+
+### Monitoring and Observability
+
+Key metrics for monitoring Unified Search:
+
+- `unified_search_requests_total`: Search request counts by type and status
+- `unified_search_request_duration_seconds`: Search request latency
+- `unified_search_index_size_bytes`: Size of search indexes
+- `unified_search_documents_total`: Number of indexed documents
+- `unified_search_indexing_duration_seconds`: Time to build/update indexes
+- `unified_search_shadow_requests_total`: Shadow traffic request counts
+- `unified_search_ring_members`: Number of active search server instances
+
+