Example: the Penvoice API

The generic recipe is in Onboard an app; this page shows it filled in for the Penvoice API.

This is the real onboarding of penvoice-api — a Go 1.26 REST API — as an example app on Ultron Infra. It is the worked, filled-in version of the generic onboarding recipe: instrument the app, containerize it, ship the image to GHCR, then deploy it from workloads/penvoice/ as an Argo Rollouts canary gated on a Prometheus metric.

End-to-end shape

flowchart LR
  Push([git push main]) --> CI[GitHub Actions]
  CI -->|build + push| GHCR[(ghcr.io/webb1es/penvoice-api)]
  Bump[bump tag in rollout.yaml] --> Argo[Argo CD]
  Argo -->|PreSync| Mig[migrations Job]
  Argo -->|sync| RO[Rollout canary]
  GHCR -.image.-> RO
  RO --> Svc[Service] --> Ing[Ingress test-api.penvoice.app]
  RO -->|/metrics| SM[ServiceMonitor] --> Prom[Prometheus]
  Prom --> AT{{AnalysisTemplate >= 0.95}}
  AT -.gate.-> RO

1. /metrics instrumentation

A metrics middleware is wired innermost, directly around the mux, so the matched route pattern (not the raw path with IDs) becomes a low-cardinality label. It exports two series via promauto — http_requests_total and http_request_duration_seconds, each labelled method, route, code:

httpRequestsTotal = promauto.NewCounterVec(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests by method, matched route, and status code.",
    },
    []string{"method", "route", "code"},
)

http_requests_total is exactly the series the canary gate later queries.

2. The Dockerfile (distroless, bundles migrate)

Two stages. The build stage cross-compiles a static binary (CGO_ENABLED=0) for arm64 — the cluster runs on an Ampere A1 node — and downloads the golang-migrate CLI (v4.19.1) so the same image can run migrations. The runtime stage is gcr.io/distroless/static-debian12:nonroot:

FROM gcr.io/distroless/static-debian12:nonroot
WORKDIR /app
COPY --from=build /out/penvoice-api /app/penvoice-api
COPY --from=build /out/migrate      /app/migrate
COPY --from=build /src/migrations   /app/migrations
EXPOSE 8080
USER nonroot:nonroot
ENTRYPOINT ["/app/penvoice-api"]

Bundling migrate and the migrations/ directory is what lets the PreSync Job (below) run with the very same image as the API.

3. CI → GHCR

.github/workflows/build.yml runs on push to main (and v* tags). It builds linux/arm64 and pushes to ghcr.io/webb1es/penvoice-api, tagging by commit SHA (type=sha → sha-<short>), latest on the default branch, and semver on tags:

env:
  IMAGE: ghcr.io/webb1es/penvoice-api
# ...
- name: Build and push (linux/arm64)
  uses: docker/build-push-action@v6
  with:
    platforms: linux/arm64
    push: true

The deployed tag is the SHA — e.g. sha-ed927c2 — which is what the manifests pin.

4. The workloads/penvoice/ manifests

The Argo CD Application (apps/penvoice.yaml) syncs workloads/penvoice/ into the penvoice namespace. Its pieces:

File	Object	Real values
namespace.yaml	Namespace	penvoice
postgres-cluster.yaml	CNPG Cluster	penvoice-pg, 1 instance, db/owner penvoice, 5Gi local-path
configmap.yaml	ConfigMap	KC_ISSUER → test-auth.webbies.dev/realms/penvoice, KC_AUDIENCE penvoice-api, CORS_ORIGINS test.penvoice.app
migrations-job.yaml	Job (PreSync hook)	runs /app/migrate ... up
rollout.yaml	Argo Rollout	2 replicas, image sha-ed927c2
service.yaml	Service	:80 → http (8080)
ingress.yaml	Ingress	test-api.penvoice.app
servicemonitor.yaml	ServiceMonitor	scrapes /metrics every 15s
analysistemplate.yaml	AnalysisTemplate	success-rate ≥ 0.95

Postgres (penvoice-pg)

A single-instance CloudNativePG Cluster. CNPG auto-creates the penvoice-pg-app Secret whose uri key supplies DB_URL. The owner bootstrap lets the app create citext/pgcrypto. (Its backup block → Backups.)

Migrations as a PreSync Job

Migrations run before the Rollout updates pods, as an Argo CD sync hook:

metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
# ...
image: ghcr.io/webb1es/penvoice-api:sha-ed927c2
command: ["/app/migrate"]
args: ["-path", "/app/migrations", "-database", "$(DB_URL)", "up"]

DB_URL comes from the CNPG secret (penvoice-pg-app → uri); the bundled migrate binary makes the image self-sufficient.

Rollout, Service, Ingress

The Rollout runs 2 replicas of ghcr.io/webb1es/penvoice-api:sha-ed927c2, pulls non-secret config via envFrom the ConfigMap, and injects DB_URL and KC_API_CLIENT_SECRET (Secret penvoice-api-kc) directly. Probes hit /readyz and /healthz. A single Service (:80 → http) fronts both stable and canary pods — sufficient for a replica-based canary. The Ingress exposes test-api.penvoice.app over Traefik with a cert-manager letsencrypt-prod certificate.

ServiceMonitor

endpoints:
  - port: http
    path: /metrics
    interval: 15s
    relabelings:
      - sourceLabels: [__meta_kubernetes_pod_label_rollouts_pod_template_hash]
        targetLabel: rollouts_pod_template_hash

The release: kube-prom-stack label is required for that Prometheus to select the monitor; the relabeling copies the Rollouts pod-template-hash onto each sample so analysis can be scoped to canary pods later.

5. The metric-gated canary

The AnalysisTemplate (penvoice-success-rate) runs as a background gate during the canary. It queries the in-cluster Prometheus for non-5xx over total and demands ≥ 95%, aborting after two sub-threshold reads:

failureLimit: 2          # abort after 2 sub-threshold reads
successCondition: "result[0] >= 0.95"
provider:
  prometheus:
    address: http://prometheus-operated.monitoring:9090
    query: |
      sum(rate(http_requests_total{namespace="penvoice",code!~"5.."}[1m]))
      /
      sum(rate(http_requests_total{namespace="penvoice"}[1m]))

The Rollout walks setWeight 25 → 50 → 75 with 60s pauses, analyzing from step 1. Pass → auto-promote; two failures → auto-rollback to the previous version, no human in the loop. To trigger a canary you bump the image tag (or the penvoice.app/redeploy annotation) in rollout.yaml and push. See Progressive delivery for the underlying mechanism.