Skip to content

Rotate a consumer token

A consumer's token authenticates it to the /v1 API. The provider stores only the SHA-256 of the token in a Secret referenced by the Consumer; that Secret key may hold several hashes, one per line, and every listed hash is accepted. Rotation uses this overlap so there is no moment without a valid token.

Rotate on a schedule, when someone with access to the token leaves, and immediately on any suspicion of exposure (in the latter case see "Emergency" below).

Runbook

The consumer-side Secret is the Secret in the consumer's cluster that holds the plaintext token (for example the token mounted by the consuming operator). Use your secret channel for it; never paste a token into chat, tickets or logs.

  1. Generate the new token and its hash.

    NEW=$(openssl rand -hex 32)
printf '%s' "$NEW" | sha256sum | cut -d' ' -f1

  2. Add the new hash as a second line in the provider-side hash Secret (the one Consumer.spec.tokenSecretRef points at), keeping the old line. Both tokens now authenticate. In a GitOps landscape this is a change to the SOPS-encrypted Secret; ad hoc:

    printf '%s\n%s\n' "<old hash>" "<new hash>" | kubectl -n <namespace> create secret generic <hash-secret> \
  --from-file=sha256=/dev/stdin --dry-run=client -o yaml | kubectl apply -f -

    (The hashes are not secret by themselves, but nothing needs to touch the disk: the list is piped in. Never write the token to a file or a shell history.)

    The provider's cache picks the change up within seconds. Check that the new token works before touching the consumer:

    curl --cacert ca.crt -H "Authorization: Bearer $NEW" https://<provider>:8443/v1/databases
# 200 with {"items": [...]}

  3. Roll the consumer-side Secret to the new token and let the consumer pick it up (restart or reload the consuming workload as that component requires). Confirm the consumer works with the new token.

  4. Watch for stragglers. While both hashes exist there is no failure to see; to be sure the old token is no longer used, confirm all consumer replicas run with the new Secret. (The provider does not record which of a consumer's hashes matched.)

  5. Remove the old hash line from the provider-side Secret, leaving only the new one. From this point the old token gets 401 unauthorized; the new one continues to work.

The behaviour of steps 2 and 5 is covered by the test TestTwoHashesDuringRotation (both tokens authenticate while both lines exist; the old one is refused after its line is removed).

Emergency (token exposed)

Do not wait for a graceful overlap: set spec.disabled: true on the Consumer (all its calls get 401 at once, existing databases are untouched), generate a new token, replace the hash Secret with the single new hash, deliver the token to the consumer, then set disabled: false.

Notes

  • Tokens should be 256-bit random values (as above). The provider does not rate-limit failed attempts; a 401 is constant-time and carries no information about why it failed, so strong tokens are the defence.
  • The same hash on two Consumers denies both. Never reuse a token across consumers.
  • Hashes may be lower or upper case. Malformed lines are ignored and logged once (without values).