Commit Graph

32 Commits

Author SHA1 Message Date
Matthew Jacobson
44972d0cd5 SQLStore: Fix Postgres dialect treating "false" migrator default as true (#69353)
* SQLStore: Fix Postgres dialect treating "false" migrator default as true

Previously, when creating a migration you could choose a default value for a new 
boolean column that looked correct but would be interpreted incorrectly by the 
Postgres dialect. For example, values such as "false" or "FALSE" would be treated 
as true by the Postgres dialect.

This refactors how migration dialects determine the Default column value for boolean
type columns. Each dialect now uses the same base code to parse the Default literal 
and panics if an unknown value is encountered.

So, now AddColumnMigration and AddTableMigration will ensure that across dialects:

- The exact same Default literals will be allowed.
- The literals are converted to equivalent defaults in their DDL.
- An error will be thrown if an invalid literal is provided.
2023-06-22 15:09:36 -04:00
Dan Cech
a6279b2d62 Database: Make dialects independent of xorm Engine (#69955)
* make dialects independent of xorm Engine

* goimports
2023-06-14 16:13:36 -04:00
Alexander Weaver
ea8c5ce929 SQLStore: Create utility for bulk inserts (#57632)
* Add batchsize to dialects

* Some intermediary progress

* Reflection-based implementation

* Validate batch size

* Implement a few simple tests

* Fix linter error

* Add debug message when falling back to default batch size
2022-11-01 13:24:32 -05:00
Emil Tullstedt
b287047052 Chore: Upgrade Go to 1.19.1 (#54902)
* WIP

* Set public_suffix to a pre Ruby 2.6 version

* we don't need to install python

* Stretch->Buster

* Bump versions in lib.star

* Manually update linter

Sort of messy, but the .mod-file need to contain all dependencies that
use 1.16+ features, otherwise they're assumed to be compiled with
-lang=go1.16 and cannot access generics et al.

Bingo doesn't seem to understand that, but it's possible to manually
update things to get Bingo happy.

* undo reformatting

* Various lint improvements

* More from the linter

* goimports -w ./pkg/

* Disable gocritic

* Add/modify linter exceptions

* lint + flatten nested list

Go 1.19 doesn't support nested lists, and there wasn't an obvious workaround.
https://go.dev/doc/comment#lists
2022-09-12 12:03:49 +02:00
Joe Blubaugh
a0f96ed4e1 SQLStore: Support Upserting multiple rows. (#52228)
This will be used to reduce write load when the alerting system writes a
large number of events.
2022-07-19 16:42:48 +08:00
Joan López de la Franca Beltran
5f1305d280 SQLStore: Add deprecation comments for breaking migrations (#49740)
* Migrator: Extend support to rename columns

* SQLStore: Add deprecation comments for breaking migrations
2022-06-03 17:42:08 +02:00
Joan López de la Franca Beltran
e43879e55d Encryption: Add support for multiple data keys per day (#47765)
* Add database migrations

* Use short uids as data key ids

* Add support for manual data key rotation

* Fix duplicated mutex unlocks

* Fix migration

* Manage current data keys per name

* Adjust key re-encryption and test

* Modify rename column migration for MySQL compatibility

* Refactor secrets manager and data keys cache

* Multiple o11y adjustments

* Fix stats query

* Apply suggestions from code review

Co-authored-by: Tania <yalyna.ts@gmail.com>

* Fix linter

* Docs: Rotate data encryption keys API endpoint

Co-authored-by: Tania <yalyna.ts@gmail.com>
2022-05-23 13:13:55 +02:00
Leonard Gram
f46038ed3a Search: sort results correctly when using postgres (#46466)
* Search: sort results correctly when using postgres

postgresql puts nulls first while both mysql and sqlite puts them last

* linting
2022-03-15 15:08:40 +01:00
Sofia Papagiannaki
d718ee1918 SQLStore: Prevent concurrent migrations (#44101)
* SQLStore: Prevent concurrent migrations

* Hide behind a feature toggle

* Configurable locking attempt timeout

* Update docs/sources/administration/configuration.md

Co-authored-by: Igor Suleymanov <radiohead@users.noreply.github.com>
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
2022-02-15 18:54:27 +02:00
Sofia Papagiannaki
8c31e25926 AlertingNG: Save alert instances (#30223)
* AlertingNG: Save alert instances

Co-authored-by: Kyle Brandt <kyle@grafana.com>

* Rename alert instance fields/columns

* Include definition title in listing alert instances

* Delete instances when deleting defintion

Co-authored-by: Kyle Brandt <kyle@grafana.com>
2021-01-18 20:57:17 +02:00
Arve Knudsen
b5379c5335 Chore: Fix SQL related Go variable naming (#28887)
* Chore: Fix variable naming

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-11-11 06:21:08 +01:00
Arve Knudsen
676d393ec9 Chore: Fix issues reported by staticcheck; enable stylecheck linter (#28866)
* Chore: Fix issues reported by staticcheck

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Apply suggestions from code review

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
2020-11-05 15:37:11 +01:00
Carl Bergquist
74d1d3c6a8 Instrumentation: Add counters and histograms for database queries (#28236)
Signed-off-by: bergquist <carl.bergquist@gmail.com>

Co-authored-by: Sofia Papagiannaki <papagian@users.noreply.github.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-10-19 20:06:12 +02:00
Sofia Papagiannaki
4937f0daab SQLStore: Run tests as integration tests (#28265)
* sqlstore: Run tests as integration tests

* Truncate database instead of re-creating it on each test

* Fix test description

See https://github.com/grafana/grafana/pull/12129

* Fix lint issues

* Fix postgres dialect after review suggestion

* Rename and document functions after review suggestion

* Add periods

* Fix auto-increment value for mysql dialect

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
2020-10-16 10:46:14 +03:00
Andrej Ocenas
e5dd7efdee Provisioning: Allows specifying uid for datasource and use that in derived fields (#23585)
* Add uid to datasource

* Fix uid passing when provisioning

* Better error handling and Uid column type change

* Fix test and strict null error counts

* Add backend tests

* Add tests

* Fix strict null checks

* Update test

* Improve tests

* Update pkg/services/sqlstore/datasource.go

Co-Authored-By: Arve Knudsen <arve.knudsen@gmail.com>

* Variable rename

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-04-20 15:48:38 +02:00
Noval Agung Prayogo
ea35ae4d1f Chore: Update the xorm dependency (#22376)
* Upgrade xorm

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-04-01 15:57:21 +02:00
Sofia Papagiannaki
7b70e7db2d AuthProxy: Optimistic lock pattern for remote cache Set (#17485)
* Implementation of optimistic lock pattern

Try to insert the remote cache key and handle integrity error

* Remove transaction

Integrity error inside a transaction results in deadlock

* Remove check for existing remote cache key

Is no longer needed since integrity constrain violations are handled

* Add check for integrity constrain violation

Do not update the row if the insert statement fails
for other than an integrity constrain violation

* Handle failing inserts because of deadlocks

If the insert statement fails because of a deadlock
try to update the row

* Add utility function for returning SQL error code

Useful for debugging

* Add logging for failing expired cache key deletion

Do not shallow it completely

* Revert "Add utility function for returning SQL error code"

This reverts commit 8e0b82c79633e7d8bc350823cbbab2ac7a58c0a5.

* Better log for failing deletion of expired cache key

* Add some comments

* Remove check for existing cache key

Attempt to insert the key without checking if it's already there
and handle the error situations

* Do not propagate deadlocks created during update

Most probably somebody else is trying to insert/update
the key at the same time so it is safe enough to ignore it
2019-06-13 15:36:09 +02:00
Marcus Efraimsson
12f3c8f9d6 fix only add column if not exists for mysql 2018-12-18 23:02:08 +01:00
Marcus Efraimsson
09b3014683 fix only create/drop database indices if not exists/exists 2018-12-18 21:47:45 +01:00
Marcus Efraimsson
d093244282 sqlstore: add support for checking if error is constraint validation error 2018-09-27 13:38:22 +02:00
Dan Cech
1dfff74da9 move database-specific code into dialects (#11884) 2018-05-10 16:54:21 +02:00
Karsten Weiss
5d95601720 Merge variable declaration with assignment (gosimple)
This fixes:
pkg/cmd/grafana-cli/commands/upgrade_all_command.go:56:3: should merge variable declaration with assignment on next line (S1021)
pkg/login/ldap.go:406:4: should merge variable declaration with assignment on next line (S1021)
pkg/services/sqlstore/migrator/dialect.go:87:2: should merge variable declaration with assignment on next line (S1021)
pkg/services/sqlstore/migrator/dialect.go:165:2: should merge variable declaration with assignment on next line (S1021)
pkg/tsdb/cloudwatch/metric_find_query_test.go:185:2: should merge variable declaration with assignment on next line (S1021)
2018-04-16 21:04:57 +02:00
Marcus Efraimsson
3d1c624c12 WIP: Protect against brute force (frequent) login attempts (#10031)
* db: add login attempt migrations

* db: add possibility to create login attempts

* db: add possibility to retrieve login attempt count per username

* auth: validation and update of login attempts for invalid credentials

If login attempt count for user authenticating is 5 or more the last 5 minutes
we temporarily block the user access to login

* db: add possibility to delete expired login attempts

* cleanup: Delete login attempts older than 10 minutes

The cleanup job are running continuously and triggering each 10 minute

* fix typo: rename consequent to consequent

* auth: enable login attempt validation for ldap logins

* auth: disable login attempts validation by configuration

Setting is named DisableLoginAttemptsValidation and is false by default
Config disable_login_attempts_validation is placed under security section
#7616

* auth: don't run cleanup of login attempts if feature is disabled

#7616

* auth: rename settings.go to ldap_settings.go

* auth: refactor AuthenticateUser

Extract grafana login, ldap login and login attemp validation together
with their tests to separate files.
Enables testing of many more aspects when authenticating a user.
#7616

* auth: rename login attempt validation to brute force login protection

Setting DisableLoginAttemptsValidation => DisableBruteForceLoginProtection
Configuration disable_login_attempts_validation => disable_brute_force_login_protection
#7616
2018-01-26 10:41:41 +01:00
Dan Cech
24d4e50343 utf8mb4 encoding (#7959)
* use utf8mb4 character set for connections to mysql

* use utf8mb4 character set for tables, shorten varchar fields used in unique indexes

* migration type to update table character set

* update table character sets

* set charset for temp_user.status

* gofmt
2017-03-28 14:34:53 +02:00
bergquist
e5c64732f1 fix(sql): Add boolstr to all dialects
closes #6116
2016-09-23 09:36:15 +02:00
Torkel Ödegaard
c7e3ed096f fix(postgres): fixes db migration issue with_credentials column for postgres, fixes #3505 2015-12-14 10:19:53 +01:00
Torkel Ödegaard
835fd383ad fix(postgres): Dashboard search is now case insensitive when using Postgres, fixes #1896 2015-08-10 20:25:01 +02:00
Torkel Ödegaard
ed68a4bb9a More work on SQL migrations 2015-02-24 17:59:21 +01:00
Torkel Ödegaard
02a89c752b Progress on database schema migration for account -> org refactor 2015-02-24 11:46:34 +01:00
Torkel Ödegaard
60541a455f Worked on search filter flag IsStarred, and updated frontend with new dashboard list panel 2015-02-04 11:35:59 +01:00
Torkel Ödegaard
0a695ba17a Final work on migration, now there is no usage of xorm table sync 2015-01-20 14:44:37 +01:00
Torkel Ödegaard
afb847acc8 a lot of work on database schema and migration setup, postgres now works, every integration test passes for all database types, only token table left to do 2015-01-20 14:15:48 +01:00