AuthN: Embed an OAuth2 server for external service authentication (#68086)

* Moving POC files from #64283 to a new branch

Co-authored-by: Mihály Gyöngyösi <mgyongyosi@users.noreply.github.com>

* Adding missing permission definition

Co-authored-by: Mihály Gyöngyösi <mgyongyosi@users.noreply.github.com>

* Force the service instantiation while client isn't merged

Co-authored-by: Mihály Gyöngyösi <mgyongyosi@users.noreply.github.com>

* Merge conf with main

Co-authored-by: Mihály Gyöngyösi <mgyongyosi@users.noreply.github.com>

* Leave go-sqlite3 version unchanged

Co-authored-by: Mihály Gyöngyösi <mgyongyosi@users.noreply.github.com>

* tidy

Co-authored-by: Mihály Gyöngyösi <mgyongyosi@users.noreply.github.com>

* User SearchUserPermissions instead of SearchUsersPermissions

* Replace DummyKeyService with signingkeys.Service

* Use user🆔<id> as subject

* Fix introspection endpoint issue

* Add X-Grafana-Org-Id to get_resources.bash script

* Regenerate toggles_gen.go
* Fix basic.go

* Add GetExternalService tests

* Add GetPublicKeyScopes tests

* Add GetScopesOnUser tests

* Add GetScopes tests

* Add ParsePublicKeyPem tests

* Add database test for GetByName

* re-add comments

* client tests added

* Add GetExternalServicePublicKey tests

* Add other test case to GetExternalServicePublicKey

* client_credentials grant test

* Add test to jwtbearer grant

* Test Comments

* Add handleKeyOptions tests

* Add RSA key generation test

* Add ECDSA by default to EmbeddedSigningKeysService

* Clean up org id scope and audiences

* Add audiences to the DB

* Fix check on Audience

* Fix double import

* Add AC Store mock and align oauthserver tests

* Fix test after rebase

* Adding missing store function to mock

* Fix double import

* Add CODEOWNER

* Fix some linting errors

* errors don't need type assertion

* Typo codeowners

* use mockery for oauthserver store

* Add feature toggle check

* Fix db tests to handle the feature flag

* Adding call to DeleteExternalServiceRole

* Fix flaky test

* Re-organize routes comments and plan futur work

* Add client_id check to Extended JWT client

* Clean up

* Fix

* Remove background service registry instantiation of the OAuth server

* Comment cleanup

* Remove unused client function

* Update go.mod to use the latest ory/fosite commit

* Remove oauth2_server related configs from defaults.ini

* Add audiences to DTO

* Fix flaky test

* Remove registration endpoint and demo scripts. Document code

* Rename packages

* Remove the OAuthService vs OAuthServer confusion

* fix incorrect import ext_jwt_test

* Comments and order

* Comment basic auth

* Remove unecessary todo

* Clean api

* Moving ParsePublicKeyPem to utils

* re ordering functions in service.go

* Fix comment

* comment on the redirect uri

* Add RBAC actions, not only scopes

* Fix tests

* re-import featuremgmt in migrations

* Fix wire

* Fix scopes in test

* Fix flaky test

* Remove todo, the intersection should always return the minimal set

* Remove unecessary check from intersection code

* Allow env overrides on settings

* remove the term app name

* Remove app keyword for client instead and use Name instead of ExternalServiceName

* LogID remove ExternalService ref

* Use Name instead of ExternalServiceName

* Imports order

* Inline

* Using ExternalService and ExternalServiceDTO

* Remove xorm tags

* comment

* Rename client files

* client -> external service

* comments

* Move test to correct package

* slimmer test

* cachedUser -> cachedExternalService

* Fix aggregate store test

* PluginAuthSession -> AuthSession

* Revert the nil cehcks

* Remove unecessary extra

* Removing custom session

* fix typo in test

* Use constants for tests

* Simplify HandleToken tests

* Refactor the HandleTokenRequest test

* test message

* Review test

* Prevent flacky test on client as well

* go imports

* Revert changes from 526e48ad45

* AuthN: Change the External Service registration form (#68649)

* AuthN: change the External Service registration form

* Gen default permissions

* Change demo script registration form

* Remove unecessary comment

* Nit.

* Reduce cyclomatic complexity

* Remove demo_scripts

* Handle case with no service account

* Comments

* Group key gen

* Nit.

* Check the SaveExternalService test

* Rename cachedUser to cachedClient in test

* One more test case to database test

* Comments

* Remove last org scope

Co-authored-by: Mihály Gyöngyösi <mgyongyosi@users.noreply.github.com>

* Update pkg/services/oauthserver/utils/utils_test.go

* Update pkg/services/sqlstore/migrations/oauthserver/migrations.go

Remove comment

* Update pkg/setting/setting.go

Co-authored-by: Gabriel MABILLE <gamab@users.noreply.github.com>

---------

Co-authored-by: Mihály Gyöngyösi <mgyongyosi@users.noreply.github.com>

pkg/services/oauthserver/models.go

@@ -0,0 +1,91 @@
+package oauthserver
+
+import (
+	"context"
+	"net/http"
+
+	"github.com/grafana/grafana/pkg/services/accesscontrol"
+	"gopkg.in/square/go-jose.v2"
+)
+
+const (
+	// TmpOrgID is the orgID we use while global service accounts are not supported.
+	TmpOrgID int64 = 1
+	// NoServiceAccountID is the ID we use for client that have no service account associated.
+	NoServiceAccountID int64 = 0
+
+	// List of scopes used to identify the impersonated user.
+	ScopeUsersSelf       = "users:self"
+	ScopeGlobalUsersSelf = "global.users:self"
+	ScopeTeamsSelf       = "teams:self"
+
+	// Supported encryptions
+	RS256 = "RS256"
+	ES256 = "ES256"
+)
+
+// OAuth2Server represents a service in charge of managing OAuth2 clients
+// and handling OAuth2 requests (token, introspection).
+type OAuth2Server interface {
+	// SaveExternalService creates or updates an external service in the database, it generates client_id and secrets and
+	// it ensures that the associated service account has the correct permissions.
+	SaveExternalService(ctx context.Context, cmd *ExternalServiceRegistration) (*ExternalServiceDTO, error)
+	// GetExternalService retrieves an external service from store by client_id. It populates the SelfPermissions and
+	// SignedInUser from the associated service account.
+	GetExternalService(ctx context.Context, id string) (*ExternalService, error)
+
+	// HandleTokenRequest handles the client's OAuth2 query to obtain an access_token by presenting its authorization
+	// grant (ex: client_credentials, jwtbearer).
+	HandleTokenRequest(rw http.ResponseWriter, req *http.Request)
+	// HandleIntrospectionRequest handles the OAuth2 query to determine the active state of an OAuth 2.0 token and
+	// to determine meta-information about this token.
+	HandleIntrospectionRequest(rw http.ResponseWriter, req *http.Request)
+}
+
+//go:generate mockery --name Store --structname MockStore --outpkg oauthtest --filename store_mock.go --output ./oauthtest/
+
+type Store interface {
+	RegisterExternalService(ctx context.Context, client *ExternalService) error
+	SaveExternalService(ctx context.Context, client *ExternalService) error
+	GetExternalService(ctx context.Context, id string) (*ExternalService, error)
+	GetExternalServiceByName(ctx context.Context, name string) (*ExternalService, error)
+	GetExternalServicePublicKey(ctx context.Context, clientID string) (*jose.JSONWebKey, error)
+}
+
+type KeyOption struct {
+	// URL       string `json:"url,omitempty"` // TODO allow specifying a URL (to a .jwks file) to fetch the key from
+	// PublicPEM contains the Base64 encoded public key in PEM format
+	PublicPEM string `json:"public_pem,omitempty"`
+	Generate  bool   `json:"generate,omitempty"`
+}
+
+type SelfCfg struct {
+	// Enabled allows the service to request access tokens for itself using the client_credentials grant
+	Enabled bool `json:"enabled"`
+	// Permissions are the permissions that the external service needs its associated service account to have.
+	Permissions []accesscontrol.Permission `json:"permissions,omitempty"`
+}
+type ImpersonationCfg struct {
+	// Enabled allows the service to request access tokens to impersonate users using the jwtbearer grant
+	Enabled bool `json:"enabled"`
+	// Groups allows the service to list the impersonated user's teams
+	Groups bool `json:"groups"`
+	// Permissions are the permissions that the external service needs when impersonating a user.
+	// The intersection of this set with the impersonated user's permission guarantees that the client will not
+	// gain more privileges than the impersonated user has.
+	Permissions []accesscontrol.Permission `json:"permissions,omitempty"`
+}
+
+// ExternalServiceRegistration represents the registration form to save new OAuth2 client.
+type ExternalServiceRegistration struct {
+	Name string `json:"name"`
+	// RedirectURI is the URI that is used in the code flow.
+	// Note that this is not used yet.
+	RedirectURI *string `json:"redirectUri,omitempty"`
+	// Impersonation access configuration
+	Impersonation ImpersonationCfg `json:"impersonation"`
+	// Self access configuration
+	Self SelfCfg `json:"self"`
+	// Key is the option to specify a public key or ask the server to generate a crypto key pair.
+	Key *KeyOption `json:"key,omitempty"`
+}