Recover a soft-deleted database¶

A soft delete is started in exactly two ways: the consumer calls DELETE /v1/databases/{externalId}, or an operator sets the annotation db.bnerd.com/delete-requested on the Database. This page is the manual procedure to get a database back within the grace period. In v1 there is no API to undelete; that is planned for a later phase.

Hold versus soft delete¶

Deleting the Database object (kubectl delete, a Flux prune, deleting the namespace or the CRD) does not delete the tenant. A Database with a deletionTimestamp but without the annotation is held:

nothing is disabled or dropped, no SQL runs at all, however long it stays like that; the tenant keeps working and the API keeps reporting it Ready with its connection;
the object stays Terminating (finalizer db.bnerd.com/tenant), with condition DeletionHeld=True, one Warning event DeletionHeld, and it is counted in dbprovider_databases_deletion_held; the alert DBProviderDeletionHeld (warning) fires after 5 minutes;
a held object on which nothing was ever created on a shard (no status.roleName) is released at once.

Reason: the provider only acts on a running tenant when asked about that database. Tooling that deletes objects in bulk must never disable a fleet of tenants.

What to do with a held Database:

It was a mistake (a Flux prune, a wrong kubectl delete): stop the source (fix the kustomization, suspend the prune). Leave the object as it is; it is harmless. It cannot be "undeleted" in Kubernetes, but it keeps working, and when its consumer later deletes it through the API the regular soft delete runs and the object goes away. If the alert is expected for a while, silence it in Alertmanager for that shard with a comment.
It was deliberate (an operator wants the database gone): set the annotation. That starts the regular soft delete below, with a fresh grace period:
```
kubectl -n <namespace> annotate database <name> db.bnerd.com/delete-requested="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
```
This annotation is also the break-glass way to delete a database without its consumer: it is the deliberate operator delete. Never remove the finalizer by hand (see the end of this page).

Deletion protection (chart default). The chart's ValidatingAdmissionPolicy (deletionProtection.enabled, default true) refuses DELETE on Database objects up front, so the hold should rarely be needed. Allowed are: the provider's own ServiceAccount (it deletes the object after the final DROP), objects that already carry the annotation, and members of deletionProtection.breakGlassGroups (default bnerd:db-provider-break-glass), e.g. kubectl --as=<you> --as-group=bnerd:db-provider-break-glass delete database <name>. That needs impersonation rights and is audit-logged; it only produces a held object anyway. The same policy refuses deleting a Shard that still holds databases (status.databaseCount > 0).

What a soft delete does¶

The provider records the request on its first observation of it (status.deleteRequestedAt) and sets status.dropAfter = first observation + deletion.gracePeriod (default 168 h). The value of the annotation db.bnerd.com/delete-requested is informational only; the clock never starts from it.
At once: ALTER ROLE ... NOLOGIN, REVOKE CONNECT ON DATABASE ... FROM <role>, the database is closed (ALTER DATABASE ... ALLOW_CONNECTIONS false), the role's sessions are terminated, and so are sessions of other tenant roles on that database (by database name). The Database is phase Deleting, its Ready condition has reason Deleting, TenantDisabled is True. The API reports Deleting with connection: null.
At status.dropAfter: DROP DATABASE ... WITH (FORCE), DROP ROLE, then the object and its credentials Secret are removed. This is irreversible.

The delete request is sticky: removing the annotation again does not cancel it, and PUT on the same externalId answers 409 database_deleting (the provider never revives a tenant that is being torn down). Both facts are covered by controller tests (TestRemovingDeleteAnnotationDoesNotRevive).

Procedure (only within the grace period)¶

Find the Database and its deadline:

kubectl -n <namespace> get databases
kubectl -n <namespace> get database <name> \
  -o jsonpath='{.status.phase} {.status.roleName} {.status.databaseName} dropAfter={.status.dropAfter}{"\n"}'

Do the steps in this order. Step 1 comes first: it is what stops the drop.

Before step 2, check that the disable has run: TenantDisabled must be True. If it is not yet (the shard was unreachable, the condition shows DisableFailed), the provider will still run the disable later and undo your manual re-enable, killing the sessions you opened:

kubectl -n <namespace> get database <name> \
  -o jsonpath='{range .status.conditions[?(@.type=="TenantDisabled")]}{.status} {.reason}{"\n"}{end}'
# expect: True Disabled

Permissions. Steps 1 and 4 write databases/status. The chart grants that only to the provider's ServiceAccount; an operator needs update/patch on databases/status in the release namespace (cluster admins have it), for example:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: db-provider-recovery, namespace: <namespace>}
rules:
- apiGroups: ["db.bnerd.com"]
  resources: ["databases", "databases/status"]
  verbs: ["get", "list", "patch", "update"]

Postpone the drop. The controller sets status.dropAfter once and never rewrites it, so moving it into the future postpones the DROP (covered by TestExtendedDropAfterPostponesDrop). Pick a date well past the work, for example 14 days from now (GNU date):
```
kubectl -n <namespace> patch database <name> --subresource=status --type=merge \
  -p "{\"status\":{\"dropAfter\":\"$(date -u -d '+14 days' +%Y-%m-%dT%H:%M:%SZ)\"}}"
```
Needs a kubectl with --subresource (1.24+) and permission to update databases/status.
Re-enable the role on the shard, connected to the shard's primary as the provisioner (or a superuser). The role and database names are status.roleName / status.databaseName (they are the same t_... string):
```
ALTER ROLE "<role>" LOGIN;
SET ROLE "<role>";                       -- provisioner holds SET on every tenant role; these are owner-only statements
GRANT CONNECT ON DATABASE "<database>" TO "<role>";
ALTER DATABASE "<database>" ALLOW_CONNECTIONS true;   -- the disable closed the database
RESET ROLE;
```
The connection limit and password were left untouched by the disable. The provider does not fight this manual change: the disable runs once, and a Ready/Deleting database is not reconciled back into NOLOGIN. (As a superuser, drop the SET ROLE / RESET ROLE lines.)
Verify access with the credentials from the Secret <database-name>-credentials (keys host, port, database, user, password, sslmode) through the shard's client endpoint. Handle the password as a secret; do not print it into shared logs.
Move the data out while access is back. The object stays in phase Deleting: GET returns connection: null and PUT returns 409 database_deleting, so the consumer cannot be pointed at the old database through the API. The supported way back to a managed database is: dump the data (pg_dump with the tenant credentials), create a new database through the API (a new externalId, or the same one after the old object is gone), restore the dump into it, and repoint the consumer. Then let the old object finish its deletion, or delete it earlier by lowering status.dropAfter to a past time. Lowering dropAfter is irreversible: the next reconcile drops the database and role at once; there is no confirmation and no way back except a restore from backup.

If the deadline has already passed and phase is Deleted (or the object is gone), the data is gone from the shard: see restore below.

Single-database restore¶

Restoring a single database from backup is a manual runbook that lives in the landscape repository of the DB cluster; this guide intentionally does not duplicate it. Restore is not an API feature in v1.

What not to do¶

Do not remove the finalizer db.bnerd.com/tenant to "rescue" an object, held or deleting: the object and its credentials Secret disappear, the role and database stay orphaned on the shard, and the consumer gets 404 (and may create a new, empty database on its next PUT).
Do not run ALTER ROLE ... LOGIN alone and assume the database is saved: without step 1 the drop still happens at the original dropAfter.