Commit Graph

47 Commits

Author SHA1 Message Date
Serge Zaitsev
694b9dfe50 Chore: Replace xorm.io/xorm imports (#104458)
* replace xorm.io/xorm imports

* replace xorm from other go.mod files

* clean up workspace

* nolint does not make sense anymore as it is not a module

* try if nolint directive helps

* use nolint:all for xorm

* add more nolints

* try to skip xorm in linter config

* exclude xorm differently

* retrigger ci
2025-05-02 17:13:01 +02:00
Peter Štibraný
e85acf047a Remove use of dialect.LikeStr (#104460)
* Fix TestIntegrationUpdatingProvisionionedDashboards* tests when running on Spanner by fixing case-insensitive search for dashboard title.

* Fix use of case-insensitive LIKE when running on Spanner.

* Fix use of LikeStr in anonstore.

* Fix use of LikeStr in ngalert/store and org/orgimpl.

* Fix use of LikeStr in queryhistory search.

* Fix use of LikeStr in serviceaccounts.

* Fix use of LikeStr in serviceaccounts.

* Fix use of LikeStr in services/team.

* Remove LikeStr method.
2025-05-02 11:23:57 +03:00
Peter Štibraný
54b8dde1e1 services/user: Fix userimpl tests when running on Spanner. (#103715)
Fix userimpl tests when running on Spanner.
2025-04-11 10:28:21 +02:00
Peter Štibraný
f7f28757f6 Skip noop migrations, instead of executing SELECT 0. (#103086)
* Skip noop migrations, instead of executing SELECT 0.
2025-03-31 15:19:32 +02:00
Peter Štibraný
f3df64b7f4 Pass BOOL values as real types (int/bool) instead of strings to SQL parameters. (#101961)
* Pass BOOL values as real types (int/bool) instead of strings to SQL parameters.

Fixes following integration tests when running with Spanner:
* TestIntegrationDataAccess
    * GetDataSourcesByType/Get_prunable_data_sources
* TestIntegrationUserAuthToken:
    * expires_correctly
    * can_properly_rotate_tokens
    *  keeps_prev_token_valid_for_1_minute_after_it_is_confirmed

* Fix more places where "true" or "false" string was passed as query parameter instead of bool value.

* Removed unit test because it brought unwanted dependencies on xorm into multiple modules.
2025-03-12 15:40:11 +01:00
Peter Štibraný
fd6a4908f1 Support Spanner's UNION syntax, which needs to be UNION DISTINCT or UNION ALL. (#101768)
* Support Spanner's UNION syntax, which needs to be UNION DISTINCT or UNION ALL.
2025-03-10 12:33:52 +01:00
Peter Štibraný
9e690bd620 Spanner support enhancements (#101634)
* Adds ability to run integration tests against spanner (by using GRAFANA_TEST_DB=spanner env variable. SPANNER_DB variable then specifies database to use: spannertest, emulator or string like /projects/<project>/instances/<instance>/databases/<db>)
* Adds feature to migration dialects to create database from a snapshot, instead of running individual migrations.
* Adds first version of Spanner snapshot, prepared from "OSS" migrations.
* Uses generated bit-reversed-positive values instead of auto_increment. (As an experiment)
2025-03-06 16:11:20 +01:00
Jeff Levin
a21a232a8e Revert read replica POC (#93551)
* Revert "chore: add replDB to team service (#91799)"

This reverts commit c6ae2d7999.

* Revert "experiment: use read replica for Get and Find Dashboards (#91706)"

This reverts commit 54177ca619.

* Revert "QuotaService: refactor to use ReplDB for Get queries (#91333)"

This reverts commit 299c142f6a.

* Revert "refactor replCfg to look more like plugins/plugin config (#91142)"

This reverts commit ac0b4bb34d.

* Revert "chore (replstore): fix registration with multiple sql drivers, again (#90990)"

This reverts commit daedb358dd.

* Revert "Chore (sqlstore): add validation and testing for repl config (#90683)"

This reverts commit af19f039b6.

* Revert "ReplStore: Add support for round robin load balancing between multiple read replicas (#90530)"

This reverts commit 27b52b1507.

* Revert "DashboardStore: Use ReplDB and get dashboard quotas from the ReadReplica (#90235)"

This reverts commit 8a6107cd35.

* Revert "accesscontrol service read replica (#89963)"

This reverts commit 77a4869fca.

* Revert "Fix: add mapping for the new mysqlRepl driver (#89551)"

This reverts commit ab5a079bcc.

* Revert "fix: sql instrumentation dual registration error (#89508)"

This reverts commit d988f5c3b0.

* Revert "Experimental Feature Toggle: databaseReadReplica (#89232)"

This reverts commit 50244ed4a1.
2024-09-25 15:21:39 -08:00
Kristin Laemmert
daedb358dd chore (replstore): fix registration with multiple sql drivers, again (#90990)
* replstore: fix registration with multiple sql drivers, again
* only compile regex once
2024-07-25 15:13:36 -04:00
Kristin Laemmert
ab5a079bcc Fix: add mapping for the new mysqlRepl driver (#89551) 2024-06-21 08:35:46 -04:00
Sofia Papagiannaki
5e88d29814 Folders: Introduce folder service function for fetching folders by org and UIDs that contain optionally the folder full path (#80716)
* Folders: Expose function for getting all org folders with specific UIDs

* Return all org folders if UIDs is empty

* Filter out not accessible folders by the user

* Modify query to optionally returning a string that contains the UIDs of all parent folders separated by slash.
2024-01-25 09:27:13 +02:00
Arati R
ca9d147a44 Give dialects control over how insert and update queries are performed (#79946)
* Refactor insert, update
* Add separate insert, update methods
* Refactor Insert, Update signatures
2024-01-11 19:55:45 +01:00
Charandas
48612063dd Grafana app platform: an aggregator cmd and package (#79948) 2024-01-08 22:33:42 +02:00
Dan Cech
67b2972052 Chore: add/update sqlstore-related helper functions (#77408)
* add/update sqlstore-related helper functions

* add documentation & tests for InsertQuery and UpdateQuery, make generated SQL deterministic by sorting columns

* remove old log line
2023-11-03 10:30:52 -04:00
Ryan McKinley
025b2f3011 Chore: use any rather than interface{} (#74066) 2023-08-30 18:46:47 +03:00
Matthew Jacobson
44972d0cd5 SQLStore: Fix Postgres dialect treating "false" migrator default as true (#69353)
* SQLStore: Fix Postgres dialect treating "false" migrator default as true

Previously, when creating a migration you could choose a default value for a new 
boolean column that looked correct but would be interpreted incorrectly by the 
Postgres dialect. For example, values such as "false" or "FALSE" would be treated 
as true by the Postgres dialect.

This refactors how migration dialects determine the Default column value for boolean
type columns. Each dialect now uses the same base code to parse the Default literal 
and panics if an unknown value is encountered.

So, now AddColumnMigration and AddTableMigration will ensure that across dialects:

- The exact same Default literals will be allowed.
- The literals are converted to equivalent defaults in their DDL.
- An error will be thrown if an invalid literal is provided.
2023-06-22 15:09:36 -04:00
Dan Cech
a6279b2d62 Database: Make dialects independent of xorm Engine (#69955)
* make dialects independent of xorm Engine

* goimports
2023-06-14 16:13:36 -04:00
Alexander Weaver
ea8c5ce929 SQLStore: Create utility for bulk inserts (#57632)
* Add batchsize to dialects

* Some intermediary progress

* Reflection-based implementation

* Validate batch size

* Implement a few simple tests

* Fix linter error

* Add debug message when falling back to default batch size
2022-11-01 13:24:32 -05:00
Emil Tullstedt
b287047052 Chore: Upgrade Go to 1.19.1 (#54902)
* WIP

* Set public_suffix to a pre Ruby 2.6 version

* we don't need to install python

* Stretch->Buster

* Bump versions in lib.star

* Manually update linter

Sort of messy, but the .mod-file need to contain all dependencies that
use 1.16+ features, otherwise they're assumed to be compiled with
-lang=go1.16 and cannot access generics et al.

Bingo doesn't seem to understand that, but it's possible to manually
update things to get Bingo happy.

* undo reformatting

* Various lint improvements

* More from the linter

* goimports -w ./pkg/

* Disable gocritic

* Add/modify linter exceptions

* lint + flatten nested list

Go 1.19 doesn't support nested lists, and there wasn't an obvious workaround.
https://go.dev/doc/comment#lists
2022-09-12 12:03:49 +02:00
Joe Blubaugh
a0f96ed4e1 SQLStore: Support Upserting multiple rows. (#52228)
This will be used to reduce write load when the alerting system writes a
large number of events.
2022-07-19 16:42:48 +08:00
Joan López de la Franca Beltran
5f1305d280 SQLStore: Add deprecation comments for breaking migrations (#49740)
* Migrator: Extend support to rename columns

* SQLStore: Add deprecation comments for breaking migrations
2022-06-03 17:42:08 +02:00
Joan López de la Franca Beltran
e43879e55d Encryption: Add support for multiple data keys per day (#47765)
* Add database migrations

* Use short uids as data key ids

* Add support for manual data key rotation

* Fix duplicated mutex unlocks

* Fix migration

* Manage current data keys per name

* Adjust key re-encryption and test

* Modify rename column migration for MySQL compatibility

* Refactor secrets manager and data keys cache

* Multiple o11y adjustments

* Fix stats query

* Apply suggestions from code review

Co-authored-by: Tania <yalyna.ts@gmail.com>

* Fix linter

* Docs: Rotate data encryption keys API endpoint

Co-authored-by: Tania <yalyna.ts@gmail.com>
2022-05-23 13:13:55 +02:00
Leonard Gram
f46038ed3a Search: sort results correctly when using postgres (#46466)
* Search: sort results correctly when using postgres

postgresql puts nulls first while both mysql and sqlite puts them last

* linting
2022-03-15 15:08:40 +01:00
Sofia Papagiannaki
d718ee1918 SQLStore: Prevent concurrent migrations (#44101)
* SQLStore: Prevent concurrent migrations

* Hide behind a feature toggle

* Configurable locking attempt timeout

* Update docs/sources/administration/configuration.md

Co-authored-by: Igor Suleymanov <radiohead@users.noreply.github.com>
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
2022-02-15 18:54:27 +02:00
Sofia Papagiannaki
8c31e25926 AlertingNG: Save alert instances (#30223)
* AlertingNG: Save alert instances

Co-authored-by: Kyle Brandt <kyle@grafana.com>

* Rename alert instance fields/columns

* Include definition title in listing alert instances

* Delete instances when deleting defintion

Co-authored-by: Kyle Brandt <kyle@grafana.com>
2021-01-18 20:57:17 +02:00
Arve Knudsen
b5379c5335 Chore: Fix SQL related Go variable naming (#28887)
* Chore: Fix variable naming

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-11-11 06:21:08 +01:00
Arve Knudsen
676d393ec9 Chore: Fix issues reported by staticcheck; enable stylecheck linter (#28866)
* Chore: Fix issues reported by staticcheck

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Apply suggestions from code review

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
2020-11-05 15:37:11 +01:00
Carl Bergquist
74d1d3c6a8 Instrumentation: Add counters and histograms for database queries (#28236)
Signed-off-by: bergquist <carl.bergquist@gmail.com>

Co-authored-by: Sofia Papagiannaki <papagian@users.noreply.github.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-10-19 20:06:12 +02:00
Sofia Papagiannaki
4937f0daab SQLStore: Run tests as integration tests (#28265)
* sqlstore: Run tests as integration tests

* Truncate database instead of re-creating it on each test

* Fix test description

See https://github.com/grafana/grafana/pull/12129

* Fix lint issues

* Fix postgres dialect after review suggestion

* Rename and document functions after review suggestion

* Add periods

* Fix auto-increment value for mysql dialect

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
2020-10-16 10:46:14 +03:00
Andrej Ocenas
e5dd7efdee Provisioning: Allows specifying uid for datasource and use that in derived fields (#23585)
* Add uid to datasource

* Fix uid passing when provisioning

* Better error handling and Uid column type change

* Fix test and strict null error counts

* Add backend tests

* Add tests

* Fix strict null checks

* Update test

* Improve tests

* Update pkg/services/sqlstore/datasource.go

Co-Authored-By: Arve Knudsen <arve.knudsen@gmail.com>

* Variable rename

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-04-20 15:48:38 +02:00
Noval Agung Prayogo
ea35ae4d1f Chore: Update the xorm dependency (#22376)
* Upgrade xorm

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-04-01 15:57:21 +02:00
Sofia Papagiannaki
7b70e7db2d AuthProxy: Optimistic lock pattern for remote cache Set (#17485)
* Implementation of optimistic lock pattern

Try to insert the remote cache key and handle integrity error

* Remove transaction

Integrity error inside a transaction results in deadlock

* Remove check for existing remote cache key

Is no longer needed since integrity constrain violations are handled

* Add check for integrity constrain violation

Do not update the row if the insert statement fails
for other than an integrity constrain violation

* Handle failing inserts because of deadlocks

If the insert statement fails because of a deadlock
try to update the row

* Add utility function for returning SQL error code

Useful for debugging

* Add logging for failing expired cache key deletion

Do not shallow it completely

* Revert "Add utility function for returning SQL error code"

This reverts commit 8e0b82c79633e7d8bc350823cbbab2ac7a58c0a5.

* Better log for failing deletion of expired cache key

* Add some comments

* Remove check for existing cache key

Attempt to insert the key without checking if it's already there
and handle the error situations

* Do not propagate deadlocks created during update

Most probably somebody else is trying to insert/update
the key at the same time so it is safe enough to ignore it
2019-06-13 15:36:09 +02:00
Marcus Efraimsson
12f3c8f9d6 fix only add column if not exists for mysql 2018-12-18 23:02:08 +01:00
Marcus Efraimsson
09b3014683 fix only create/drop database indices if not exists/exists 2018-12-18 21:47:45 +01:00
Marcus Efraimsson
d093244282 sqlstore: add support for checking if error is constraint validation error 2018-09-27 13:38:22 +02:00
Dan Cech
1dfff74da9 move database-specific code into dialects (#11884) 2018-05-10 16:54:21 +02:00
Karsten Weiss
5d95601720 Merge variable declaration with assignment (gosimple)
This fixes:
pkg/cmd/grafana-cli/commands/upgrade_all_command.go:56:3: should merge variable declaration with assignment on next line (S1021)
pkg/login/ldap.go:406:4: should merge variable declaration with assignment on next line (S1021)
pkg/services/sqlstore/migrator/dialect.go:87:2: should merge variable declaration with assignment on next line (S1021)
pkg/services/sqlstore/migrator/dialect.go:165:2: should merge variable declaration with assignment on next line (S1021)
pkg/tsdb/cloudwatch/metric_find_query_test.go:185:2: should merge variable declaration with assignment on next line (S1021)
2018-04-16 21:04:57 +02:00
Marcus Efraimsson
3d1c624c12 WIP: Protect against brute force (frequent) login attempts (#10031)
* db: add login attempt migrations

* db: add possibility to create login attempts

* db: add possibility to retrieve login attempt count per username

* auth: validation and update of login attempts for invalid credentials

If login attempt count for user authenticating is 5 or more the last 5 minutes
we temporarily block the user access to login

* db: add possibility to delete expired login attempts

* cleanup: Delete login attempts older than 10 minutes

The cleanup job are running continuously and triggering each 10 minute

* fix typo: rename consequent to consequent

* auth: enable login attempt validation for ldap logins

* auth: disable login attempts validation by configuration

Setting is named DisableLoginAttemptsValidation and is false by default
Config disable_login_attempts_validation is placed under security section
#7616

* auth: don't run cleanup of login attempts if feature is disabled

#7616

* auth: rename settings.go to ldap_settings.go

* auth: refactor AuthenticateUser

Extract grafana login, ldap login and login attemp validation together
with their tests to separate files.
Enables testing of many more aspects when authenticating a user.
#7616

* auth: rename login attempt validation to brute force login protection

Setting DisableLoginAttemptsValidation => DisableBruteForceLoginProtection
Configuration disable_login_attempts_validation => disable_brute_force_login_protection
#7616
2018-01-26 10:41:41 +01:00
Dan Cech
24d4e50343 utf8mb4 encoding (#7959)
* use utf8mb4 character set for connections to mysql

* use utf8mb4 character set for tables, shorten varchar fields used in unique indexes

* migration type to update table character set

* update table character sets

* set charset for temp_user.status

* gofmt
2017-03-28 14:34:53 +02:00
bergquist
e5c64732f1 fix(sql): Add boolstr to all dialects
closes #6116
2016-09-23 09:36:15 +02:00
Torkel Ödegaard
c7e3ed096f fix(postgres): fixes db migration issue with_credentials column for postgres, fixes #3505 2015-12-14 10:19:53 +01:00
Torkel Ödegaard
835fd383ad fix(postgres): Dashboard search is now case insensitive when using Postgres, fixes #1896 2015-08-10 20:25:01 +02:00
Torkel Ödegaard
ed68a4bb9a More work on SQL migrations 2015-02-24 17:59:21 +01:00
Torkel Ödegaard
02a89c752b Progress on database schema migration for account -> org refactor 2015-02-24 11:46:34 +01:00
Torkel Ödegaard
60541a455f Worked on search filter flag IsStarred, and updated frontend with new dashboard list panel 2015-02-04 11:35:59 +01:00
Torkel Ödegaard
0a695ba17a Final work on migration, now there is no usage of xorm table sync 2015-01-20 14:44:37 +01:00
Torkel Ödegaard
afb847acc8 a lot of work on database schema and migration setup, postgres now works, every integration test passes for all database types, only token table left to do 2015-01-20 14:15:48 +01:00