Skip to content

Backup & restore

On Ultron Infra, every Postgres cluster (each app’s database, plus the platform Keycloak) is managed by CNPG and backs up to Oracle Object Storage over the S3-compat API: a nightly base backup, continuous WAL archiving, and a 30-day PITR window.

One bucket, per-cluster folders:

ClusterNamespaceBucket folder
App Postgres<app><bucket>/<cluster>
Keycloak Postgreskeycloak<bucket>/<cluster>

Credentials live in out-of-band Secrets (<app>-pg-backup-creds, keycloak-pg-backup-creds) — see Reference. These are not in Git; recreate them on rebuild.

The nightly run is a ScheduledBackup. To take an on-demand base backup, apply a Backup resource targeting the cluster:

apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
name: <app>-pg-manual-$(date +%s)
namespace: <app>
spec:
cluster:
name: <app>-pg
Terminal window
kubectl apply -f backup.yaml
kubectl get backup -n <app> -w # watch phase -> completed

WAL archiving + the base backups let you restore to any moment in the last 30 days. You don’t restore in place — you create a new cluster that bootstraps by recovering from the object store:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: <app>-pg-restore
namespace: <app>
spec:
instances: 1
bootstrap:
recovery:
source: <app>-pg
recoveryTarget:
targetTime: "2026-06-09 14:30:00+00" # within the 30-day window
externalClusters:
- name: <app>-pg
# ...barman object store config pointing at <bucket>/<app>-pg

Once the new cluster is healthy and verified, repoint the app at it (the canary makes that cutover safe — see Operate rollouts).

S3-compat gotchas (required for backups to work at all)

Section titled “S3-compat gotchas (required for backups to work at all)”

Oracle’s S3-compatible endpoint needs two non-obvious settings, already applied on every cluster:

  • Disable checksum trailers. botocore ≥ 1.36 sends checksum trailers that Oracle returns NotImplemented for. Set AWS_REQUEST_CHECKSUM_CALCULATION=when_required and AWS_RESPONSE_CHECKSUM_VALIDATION=when_required.
  • Set the region: AWS_DEFAULT_REGION=<region>.