Secret rotation runbook

Rotate sandbox and live credentials independently. Never copy a live secret into staging, and never put a secret value in source control, a URL, a log, or a command-line argument.

Rotation matrix

Secret	Scope	Safe order	Old value can be removed when
Tenant API key	One tenant integration	Create replacement → deploy consumer → verify → revoke old	The consumer is using the replacement and the old key is revoked
Webhook signing secret	One tenant endpoint	Teach receiver both → update Gateway → verify → remove old receiver secret	The delivery/retry window has drained
VERIFIER_KEY_ENCRYPTION_SECRET	One Gateway environment	Deploy current=new with previous=old → exercise/read records → verify → remove previous	All required records have been read/rewrapped and rollback window has closed
JAR verifier signing key	One Gateway environment	Mint → activate → verify JWKS → wait → revoke old kid	Existing request objects and caches can no longer reference the old key

Tenant API key

For a normal integration, create a second restricted key. This avoids the immediate invalidation that comes with rotating the same key ID:

const replacement = await id.apiKeys.create({
  name: "production-worker-2026-09",
  scopes: ["session:create", "session:read"],
  expires_at: "2027-09-21T00:00:00Z",
});

// Store replacement.api_key in the consumer's secret manager; never log it.

Deploy the consumer with the replacement key and run one controlled session. After the new key has authenticated successfully, revoke the old key:

await id.apiKeys.revoke(oldKeyId);

For an emergency replacement where the key ID must remain stable, use id.apiKeys.rotate(keyId). The returned plaintext is shown once and the old secret stops working immediately, so deploy the result in the same maintenance window.

Lifecycle audit events

The Gateway records api_key.created, api_key.rotated, and api_key.revoked in the tenant's management audit stream. Events include the key ID as a one-way hash and safe lifecycle metadata, but never the plaintext API key or its hash. Audit-write failures are reported operationally without turning a successful key mutation into a failed request.

Webhook signing secret

The Gateway signs each delivery with the current tenant endpoint secret. The receiver must accept both secrets before the Gateway is changed:

1. Generate a new whsec_... value in the secret manager.
2. Deploy the receiver with old and new verification secrets. Keep the new secret first for new deliveries, but retain the old one for verification during the overlap window.
3. Update the Gateway endpoint from a trusted server:

await id.webhooks.endpoints.upsert({
  url: process.env.DLBR_ID_WEBHOOK_URL!,
  signing_secret: process.env.DLBR_ID_WEBHOOK_SECRET_NEXT!,
});

4. Run a controlled test session and verify the raw body with id.webhooks.constructEvent() and the new secret.
5. Keep the old receiver secret until queued retries and the configured webhook replay window have drained, then remove it.

The Gateway stores the endpoint secret encrypted at rest. Updating the endpoint is a replacement, not a dual-signing period; the dual verification window belongs at the receiver.

Gateway encryption secret

VERIFIER_KEY_ENCRYPTION_SECRET protects private verifier keys, certificates, RP access keys, and encrypted webhook endpoint secrets. The Gateway accepts a bounded current/previous pair and rewrites records when they are read.

For each environment, deploy both values together through the secret manager:

VERIFIER_KEY_ENCRYPTION_SECRET=<new secret>
VERIFIER_KEY_ENCRYPTION_SECRET_PREVIOUS=<old secret>

After deployment:

1. Fetch /.well-known/jwks.json and run a controlled JAR/session request.
2. Exercise the registered RP and webhook paths so active encrypted records are read and re-encrypted with the new secret.
3. Check error logs and the controlled session/webhook result.
4. Keep the old secret as VERIFIER_KEY_ENCRYPTION_SECRET_PREVIOUS for the agreed rollback window. Remove it only after the rewrap/read coverage is complete.

Do not delete the old secret before verification. If rollback is necessary, restore the old value as current and keep the new value as previous until records written during the failed rollout are no longer needed.

JAR verifier signing key

This is separate from encryption-secret rotation. Use the repository script so the public JWKS contains the active key while the old key remains available during rollover:

cd apps/gateway
node scripts/rotate-verifier-key.ts --list --env staging --remote
node scripts/rotate-verifier-key.ts --mint --activate --env staging --remote
node scripts/rotate-verifier-key.ts --list --env staging --remote

The script requires VERIFIER_KEY_ENCRYPTION_SECRET in the operator environment. Verify /.well-known/jwks.json and a controlled JAR request before revoking the old kid:

node scripts/rotate-verifier-key.ts --revoke <old-kid> --env staging --remote

Never revoke the active kid; the script refuses that operation.

Preflight and evidence

Record only non-secret metadata in the rotation log:

• environment and deployment version;
• key ID or verifier kid (never the plaintext secret);
• start/completion timestamps;
• controlled request ID and livemode result;
• webhook delivery verification result;
• operator and rollback deadline.

The same sequence applies to production after replacing --env staging with the production environment and using sk_live_ credentials.