Configure shards and consumers¶

Fleet configuration is declarative: Shard and Consumer objects live in your landscape repository and are reconciled by Flux. Adding a shard or a consumer is a reviewed merge request, not an API call. Shard and Consumer are cluster-scoped; the group is db.bnerd.com/v1alpha1.

Shards¶

A Shard describes one Percona PG cluster the provider may place databases into.

apiVersion: db.bnerd.com/v1alpha1
kind: Shard
metadata:
  name: shard01
spec:
  engine: postgres            # only value in v1 (default)
  accepting: true             # false = drain (below)
  maxDatabases: 200           # soft limit, see below
  perconaCluster:
    namespace: databases
    name: pg-shard01
  admin:                      # how the PROVIDER reaches the shard to run DDL
    serviceName: pg-shard01-primary   # the in-cluster PRIMARY Service, not pgBouncer
    port: 5432                        # default
    sslmode: require                  # default
    secretRef:
      namespace: databases            # the release namespace, or listed (with this name) in the chart value shardSecrets
      name: pg-shard01-pguser-provisioner
      userKey: user                   # defaults
      passwordKey: password
  client:                     # what CONSUMERS are told to connect to
    host: 192.0.2.200         # the pgBouncer behind the private load balancer; hostname or IPv4 (below)
    port: 5432
    sslmode: require

Two endpoints, on purpose:

admin endpoint = the shard's primary Service inside the cluster. DDL never goes through pgBouncer, and the primary Service follows a Patroni failover; the provider retries on connection loss.
client endpoint = the pgBouncer load balancer. This is the host, port and sslmode written into the credentials returned to consumers.

Changing client is propagated to the endpoint keys of the credentials Secrets of that shard's databases right away (the Database controller watches Shards), and never touches Postgres.

client.host must be an RFC-1123 hostname without a trailing dot (letters, digits, hyphens; labels of at most 63 characters; at most 253 characters) or a dotted-quad IPv4 address without leading zeros. IPv6 literals are refused. The CRD rejects anything else, and the API re-checks it before handing out a connection (an invalid host gives 503 credentials_unavailable): consumers write the host into shell probes.

admin.sslmode and client.sslmode accept disable, allow, prefer, require (default), verify-ca and verify-full. For the admin connection, verify-ca / verify-full work only if the shard's server certificate chains to a CA in the provider image's system trust store: v0.1.0 has no setting for a custom CA (sslrootcert), so a Percona-generated CA fails the handshake and the Shard reports Unreachable. require encrypts but does not authenticate the server; the admin traffic stays inside the cluster.

The `provisioner` role¶

The admin Secret holds the credentials of a role named provisioner (the name is not enforced by the provider; the role's attributes are what matter):

LOGIN CREATEDB CREATEROLE, no superuser. It can therefore only manage roles it created itself and cannot touch postgres, pgexporter or the Percona operator's own users.
On PostgreSQL 16 and later the creator of a role receives only ADMIN OPTION on it, not SET. The provider grants itself SET on every tenant role it creates (GRANT <role> TO CURRENT_USER WITH SET TRUE, INHERIT FALSE) before CREATE DATABASE ... OWNER <role>, and uses SET ROLE <tenant> only for owner-only statements (REVOKE ... FROM PUBLIC, GRANT CONNECT, ALTER DATABASE ... ALLOW_CONNECTIONS, DROP DATABASE). A new database is created with ALLOW_CONNECTIONS false and opened only after PUBLIC access is revoked, so no other role can connect in between. INHERIT FALSE means the provisioner does not gain the tenant's privileges, e.g. it cannot read tenant data.
The credential never leaves the DB cluster.

Capacity: `maxDatabases` is a soft limit¶

Placement counts Database objects from the provider's informer cache and picks the accepting, ready shard with the fewest databases that is below maxDatabases. The Database controller reconciles up to 4 databases in parallel, so under a burst of simultaneous creates a shard may overshoot maxDatabases by a few (up to about 4). Size the limit with that in mind; do not use it as a hard quota. (The CRD requires maxDatabases >= 1. A value of 0 would be treated by the near-capacity alert as "no limit configured", i.e. the alert is disabled for that shard.)

A placement is persisted before any SQL runs and is never moved afterwards. If no shard is accepting, ready and below its limit, the Database becomes phase Failed (reason NoCapacity) and is retried every 5 minutes; the API reports phase Failed.

Draining a shard: `accepting: false`¶

Set spec.accepting: false to stop new placements on a shard. Existing databases keep working and keep being managed (including deletes). There is no move between shards in v1; a shard empties as its databases are deleted. With deletionProtection.enabled (default) the chart's admission policy refuses deleting a Shard whose status.databaseCount is above 0 (break-glass groups excepted); its databases would otherwise be left with ShardNotFound and could be neither disabled nor dropped.

Shard `Ready` condition¶

kubectl get shard shows Accepting, Databases and Max. The status conditions explain health; the provider pings the admin connection every 60 seconds and on Shard or Database changes. The admin Secret is read by name, uncached (the provider has get on exactly the names in the chart value shardSecrets, no list/watch); there is no watch on it, so a rotated, deleted or broken admin Secret shows on the Shard within 60 seconds (the next periodic check).

Ready reason	Meaning
`Ok` (status True)	admin connection healthy
`AdminAuthFailed`	Postgres rejected the provisioner credentials (SQLSTATE 28P01 / 28000); condition `AdminAuthFailed` is also True
`AdminSecretInvalid`	admin Secret missing, not readable (RBAC: its name is not in the chart value `shardSecrets`), or lacks the user/password key
`Unreachable`	anything else (network, DNS, TLS, timeout); condition `Unreachable` is also True

Messages carry a sanitized error class only, never driver text or credentials. A shard that is not Ready receives no new placements.

Consumers¶

A Consumer is one calling cluster. It can see and act on its own databases only.

apiVersion: db.bnerd.com/v1alpha1
kind: Consumer
metadata:
  name: consumer-a
spec:
  tokenSecretRef:
    namespace: databases        # MUST be the release namespace
    name: consumer-a-token
    key: sha256
  maxDatabases: 50              # quota: set it on EVERY consumer (see below)
  disabled: false               # true = every call of this consumer gets 401

Token. The consumer's bearer token is a random 256-bit value that you generate; the provider never stores or sees it in clear. Only its SHA-256 (lowercase or uppercase hex, 64 characters) is stored, in the Secret key referenced by tokenSecretRef:

TOKEN=$(openssl rand -hex 32)                       # deliver to the consumer through your secret channel, never into chat or logs
printf '%s' "$TOKEN" | sha256sum | cut -d' ' -f1    # the value for the hash Secret

kubectl -n databases create secret generic consumer-a-token \
  --from-literal=sha256="<the hash>"

(Hash exactly the token as sent, with no trailing newline: hence printf '%s'.)

The Secret key may hold several hashes, one per line. Every listed hash authenticates the consumer; this is what makes rotation possible without downtime. Empty lines and surrounding whitespace are ignored; lines that are not a hex SHA-256 are ignored and logged once.

Rules, all of which end in the same generic 401:

tokenSecretRef.namespace must be the release namespace. The provider has no access to Secrets elsewhere; a ref pointing outside never authenticates (logged once per Consumer generation).
A Consumer with disabled: true never authenticates. Use it to cut a consumer off at once.
The same hash configured on two Consumers denies both (a misconfiguration is never resolved by picking one).
A missing Secret or key means that Consumer cannot authenticate.

Set maxDatabases on every Consumer. Without it a consumer (or a bug or a leaked token on its side) can create databases until every shard is full, which blocks all other consumers. The field is optional in the CRD and there is deliberately no built-in default; size it per consumer from its expected tenant count plus head room, and raise it by a reviewed change. maxDatabases on a Consumer counts that consumer's databases that are not yet Deleted (databases in Deleting still count). Exceeding it gives 429 quota_exceeded. The count is exact within one replica; simultaneous creates on two replicas may overshoot by one.

kubectl get consumer shows Disabled and the Databases count.

Database sizes: `sizeClasses`¶

sizeClass in a PUT selects a connection limit per role from the chart value sizeClasses. Omitted or empty: connectionLimit.default. Unknown name: 400 unknown_size_class. The size class is applied at create time only; the body of a PUT for an existing database is not applied and resize is not part of the API in v1.

Changing sizeClasses or connectionLimit.default does not touch existing databases. A Ready database runs SQL only for its own inputs (its credentials, or its own spec.sizeClass changed by an operator, which runs exactly one ALTER ROLE ... CONNECTION LIMIT and never re-enables login). When the install config now resolves a different limit than the one on the shard (status.appliedConnectionLimit), or the database's class is no longer configured, the database stays Ready and gets condition ConnectionLimitDrift=True (reason ConfigChanged or UnknownSizeClass), counted in dbprovider_databases_connection_limit_drift. To apply a new limit to a database deliberately, change that Database's spec.sizeClass (to a class with the wanted limit). The class is validated against the config only when a database is created.

Database status and reasons¶

kubectl -n <namespace> get databases shows phase, shard and consumer. Phases: Provisioning, Ready, Deleting, Deleted, Failed. The Ready condition's reason explains non-ready states:

Reason	Meaning
`Provisioning`	placed, SQL pending
`Provisioned`	tenant exists on the shard (Ready=True)
`NoCapacity`	no accepting, ready shard below `maxDatabases`; retried every 5 min
`InvalidSizeClass`	the class is not in `sizeClasses` (create path only)
`ShardNotFound`	the placed shard object is gone
`AdminUnavailable`	no usable admin connection to the shard
`EnsureFailed`	create/converge SQL failed (sanitized class in the message)
`CredentialsSecretConflict`	`<name>-credentials` exists but is not owned by this Database, or is inconsistent
`Deleting`	soft delete in progress
`SetLimitFailed`	changing the connection limit after a `spec.sizeClass` change failed
`DisableFailed` / `DropFailed`	the disable or the final drop failed; retried with backoff, finalizer held

Further conditions: DeletionHeld=True (the object was deleted without the delete annotation and is held, see Recover a soft-deleted database), ConnectionLimitDrift=True (see sizes), TenantDisabled / TenantDropped (soft delete progress).

Every SQL operation is bounded: 2 minutes per operation in the provider, and statement_timeout=60s / lock_timeout=10s on the admin sessions; a hung shard surfaces as ... failed: timeout.

The credentials live in the Secret <database-name>-credentials in the release namespace (keys host, port, database, user, password, sslmode). Read them only when you must; they are also returned by the API to the owning consumer.

Monitoring and audit¶

Metrics are served by the manager on the metrics port (:8080/metrics, plain HTTP, enable the ServiceMonitor with metrics.serviceMonitor.enabled=true):

Metric	Labels	Meaning
`dbprovider_databases`	shard, phase	databases per shard and phase (`shard="unplaced"` = not yet placed)
`dbprovider_shard_ready`	shard	1 if the Shard's Ready condition is True
`dbprovider_shard_databases` / `dbprovider_shard_max_databases`	shard	slots in use / `spec.maxDatabases`
`dbprovider_database_provisioning_oldest_seconds`	shard	age of the oldest Database still Provisioning
`dbprovider_databases_deletion_held`	shard	Databases deleted without the delete annotation (held, no SQL)
`dbprovider_databases_connection_limit_drift`	shard	Ready Databases whose applied connection limit differs from the install config
`dbprovider_reconcile_errors_total`	controller, reason	failed reconciles (reason = condition reason, else `Other`)
`dbprovider_reconcile_conflicts_total`	controller	write conflicts, requeued with backoff and not counted as errors (a steady rate on one controller = an object fighting another writer)
`dbprovider_api_requests_total`	method, route, code	API requests by route template
`dbprovider_api_request_duration_seconds`	method, route	API latency

No metric carries consumer, externalId or role names.

Alerts (chart rules/alerts.yaml): DBProviderDown (critical, 10 m), DBProviderShardNotReady (critical, 5 m), DBProviderDatabaseStuckProvisioning (warning, over 15 min), DBProviderShardNearCapacity (warning, at least 90 % of maxDatabases for 10 m; disabled for a shard whose max is 0), DBProviderReconcileErrors (warning, 15 m), DBProviderDatabaseFailed (warning, phase Failed for 15 m), DBProviderDeletionHeld (warning, a held Database for 5 m).

Where the PrometheusRule goes. An agent-only DB cluster evaluates no rules. Apply the PrometheusRule at the central Prometheus (the one fed by remote_write), not in the DB cluster. Extract it with Helm and ship the result with that Prometheus' own configuration:

helm template db-provider charts/bnerd-database-provider \
  --namespace <namespace> \
  --set tls.secretName=<tls-secret-name> \
  --set prometheusRule.enabled=true \
  --set prometheusRule.cluster=<value of the `cluster` external label of this DB cluster> \
  --show-only templates/prometheusrule.yaml > prometheusrule.yaml

Set prometheusRule.cluster (per-cluster absence): every alert then selects only that cluster, and DBProviderDown detects that cluster's provider vanishing for good. Without it the rules match all clusters and per-cluster absence is not detected (the chart renders a warning). One rule release per DB cluster; decommissioning a cluster means removing its rule release. prometheusRule.jobSelector defaults to the ServiceMonitor's job name; set it when your agent relabels job. jobSelector and cluster must not contain ", \ or a newline (the render fails).

Audit trail. Kubernetes Events on the Database (kubectl -n <namespace> describe database <name>): Created, Ready, DeleteRequested, Disabled, Dropped, plus Warnings for failures and DeletionHeld (reason = condition reason, sanitized error class only). Messages carry consumer, externalId and shard; never passwords, hashes or tokens. Logs are structured, carry consumer and database identifiers, and never contain credentials or tokens. Events are retained by Kubernetes for a short time only; forward them if you need a long-term audit record.

Configure shards and consumers¶

Shards¶

The provisioner role¶

Capacity: maxDatabases is a soft limit¶

Draining a shard: accepting: false¶

Shard Ready condition¶