sharang
1fd03592c0
ci / shared (push) Successful in 11s
Same 4 VMs / flavors / SLA. 3 Orca clusters (was 1), 1 cluster per plane: breakpilot-edge (Identity + Infra), breakpilot-control (Control), breakpilot-app (App, prod + stage VMs). vm-data → vm-app-prod; stage VM → vm-app-stage. Data plane → App plane. KC co-tenant on vm-edge with JVM heap pinned. Companions: PR #6 here (impl plan) + orca-platform PR #6 (cluster manifests).
54 KiB
54 KiB
Infrastructure Specification
Status: Locked Topology Authors: Sharang, Benjamin Date: 2026-05-11 (topology lock: 2026-05-18; cluster split: 2026-06-30) Companion docs: PLATFORM_ARCHITECTURE.md, IMPLEMENTATION_PLAN.md, COST_PLAN.md Cloud provider: SysEleven Cloud Services (DUS2, OpenStack) — multi-VM rollout gated on legal entity
2026-06-30 cluster-split decision. The 4 VMs from the May 18 lock are unchanged in count and flavor. What changes is how they're organized: one Orca cluster per plane, three cluster repos (
platform/breakpilot-edge,platform/breakpilot-control,platform/breakpilot-app). Identity and Infra are co-tenant onvm-edgeinsidebreakpilot-edge(single-VM core; KC JVM heap pinned so it cannot starve PowerDNS / Infisical). The "Data plane" is renamed to App plane to reflect that it carries product workloads, not just data stores.vm-datais renamedvm-app-prod, and the standalonestageVM is renamedvm-app-stage— both live in the samebreakpilot-appcluster. Multi-VM rollout is gated on the legal entity being established so we can sign SysEleven business contracts; until then, single-VM ops continues via~/workspace/orca-infra. SeeIMPLEMENTATION_PLAN.md §1.1 / M1.2andplatform/orca-platform/clusters/README.mdfor the migration mechanics. The SLA targets in §6 / §7 below are unaffected — they apply at the plane level regardless of which cluster a plane lives in.
1. VM Inventory
Four billable VMs across three Orca clusters. Dev runs entirely on developer laptops via docker-compose.
┌──────────────────┬────────────────────┬──────────┬────────────────────────┬───────────┬─────────────────┐
│ Name │ Cluster │ Env │ SysEleven flavor │ Public IP │ Planes owned │
├──────────────────┼────────────────────┼──────────┼────────────────────────┼───────────┼─────────────────┤
│ vm-edge │ breakpilot-edge │ prod │ m2.small (2v / 8 GB) │ YES (1) │ Identity + Infra│
│ vm-control │ breakpilot-control │ prod │ m2.medium (4v / 16 GB) │ No │ Control │
│ vm-app-prod │ breakpilot-app │ prod │ m2.medium (4v / 16 GB) │ No │ App (prod) │
│ vm-app-stage │ breakpilot-app │ stage │ m2.small (2v / 8 GB) │ YES (1) │ App (stage) │
│ (dev) │ — │ dev │ local docker-compose │ n/a │ all (in-memory) │
└──────────────────┴────────────────────┴──────────┴────────────────────────┴───────────┴─────────────────┘
Total compute: 48 GiB-RAM, 12 vCPU. Monthly compute net: €192 (36M upfront) / €295 (12M) / €435 (On-Demand). See COST_PLAN.md for the full three-mode table. The cluster split does not change the per-VM bill (same flavors); the additional cost is operational (3 cluster repos to keep CI green in) and is offset by the independent failure-domain wins.
Why this topology and not the previous 7-VM layout
The earlier draft proposed one VM per service group (vm-gateway, vm-identity, vm-secrets, vm-ops, vm-control, vm-certifai, vm-compliance). That gave maximum failure isolation but cost 132 GiB-RAM stage+prod. At 5 customers the isolation is unused — every VM ran at <10% utilisation. The locked topology buys back failure isolation incrementally as load grows (see §13 Growth Trajectory).
Critical isolations preserved even at 4 VMs:
- vm-edge isolates identity from app workloads. Keycloak JVM has its own page cache + heap pin (
-Xmx1500m) so ERPNext or product background jobs on a different VM cannot starve token issuance — and PowerDNS / Infisical on the same VM cannot be starved by KC either. - vm-app-prod isolates product databases from stateless services. All app-plane DBs share one host, but they're walled off from the portal + ERPNext + Stalwart competing on vm-control.
- vm-app-stage runs the app plane only. It calls prod Keycloak (
breakpilot-edge) + prod Tenant Registry (breakpilot-control) undertenant.kind = stagerather than mirroring those services. Stage and prod live in the samebreakpilot-appOrca cluster on different VMs — same config, different physical workloads.
2. Service-to-VM Mapping
Each VM is owned by one Orca cluster (see §1). Service manifests live in that cluster's repo at platform/breakpilot-<name>/services/....
vm-edge (cluster: breakpilot-edge, prod, m2.small 8 GB, public IP)
├── orca-proxy (Orca-managed; wildcard TLS terminator)
├── powerdns-auth (Orca-managed; authoritative DNS for breakpilot.com)
├── keycloak-26 (Orca-managed; JVM, ~1.5 GB heap; repurposed from CERTifAI KC at M2.1)
├── postgres-keycloak (Orca-managed; dedicated PG instance for Keycloak only)
├── infisical (Orca-managed)
├── postgres-infisical (Orca-managed; dedicated PG instance for Infisical only)
├── redis-infisical (Orca-managed; ephemeral)
└── gitea (Orca-managed; SQLite backend to avoid a third PG)
vm-control (cluster: breakpilot-control, prod, m2.medium 16 GB)
├── customer-portal (Orca-managed; Next.js)
├── tenant-registry (Orca-managed; Go)
├── orca-controller (Orca core process; NOT a managed container)
├── erpnext (Orca-managed; Frappe bench)
├── frappe-hd (same bench as ERPNext)
├── mariadb (Orca-managed; for ERPNext)
├── redis-erpnext (Orca-managed)
└── stalwart-mail (Orca-managed; SMTP/IMAP/JMAP on mail.breakpilot.com)
vm-app-prod (cluster: breakpilot-app, prod, m2.medium 16 GB)
├── certifai-dashboard (Orca-managed)
├── mongodb (Orca-managed)
├── litellm (Orca-managed)
├── backend-compliance (Orca-managed)
├── ai-compliance-sdk (Orca-managed)
├── admin-compliance (Orca-managed)
├── postgres-app (Orca-managed; schemas: tenant_registry, compliance)
├── qdrant (Orca-managed)
└── minio (Orca-managed)
vm-app-stage (cluster: breakpilot-app — SAME cluster as vm-app-prod, stage, m2.small 8 GB, public IP)
├── orca-proxy (light; only routes to stage app)
├── customer-portal (NEW VERSION under test)
├── tenant-registry (placeholder; stage actually calls PROD tenant-registry — manifest exists for parity, see §5 isolation rules)
├── certifai-dashboard (NEW VERSION under test)
├── backend-compliance (NEW VERSION under test)
├── ai-compliance-sdk (NEW VERSION under test)
├── admin-compliance (NEW VERSION under test)
├── litellm (light; same image as prod)
├── postgres-app-stage (ephemeral; lives entirely on vm-app-stage)
├── mongodb-stage (ephemeral)
└── qdrant-stage (ephemeral, tiny corpus)
Calls OUT to prod:
→ auth.breakpilot.com (PROD Keycloak in breakpilot-edge, under stage client_id, tenant.kind = "stage")
→ registry.breakpilot.com (PROD tenant-registry in breakpilot-control, read-only for stage tenants)
→ mail.breakpilot.com (PROD Stalwart in breakpilot-control, recipient filter forces +stage@ only)
→ Polar SANDBOX webhook URL (NEVER prod Polar)
→ no calls to prod Postgres-app, MariaDB, MongoDB
Stage isolation rules (enforced at the platform, not in product code)
| Risk | Enforcement mechanism | Owner |
|---|---|---|
| Stage writes to prod database | Infisical scope: stage app only gets /stage/* secrets. Prod DB credentials never reach stage. |
Infra plane |
| Stage emails real customers | Stalwart accept-rule: drop if recipient does not match *+stage@*. |
Control plane (Stalwart config) |
| Stage triggers real Polar charges | Stage env points POLAR_API_URL to sandbox. Prod Polar webhook secret never on stage. |
Control plane |
| Stage Keycloak JWT used in prod | stage_client_id issued only by Keycloak; prod services reject JWTs with this aud. |
Identity plane |
| Stage load DOSes prod Keycloak | Keycloak rate-limit per client_id; stage limited to 60 req/s. | Identity plane |
3. Network Topology
INTERNET
│
(breakpilot.com — authoritative on vm-edge PowerDNS;
stage.breakpilot.com — authoritative same zone)
│
┌─────────────┴─────────────┐
│ │
┌───────▼────────┐ ┌────────▼─────────┐
│ vm-edge │ │ stage │
│ (public IP) │ │ (public IP) │
│ │ │ │
│ orca-proxy ────┤ │ orca-proxy │
│ powerdns │ │ portal-new │
│ keycloak │◄────────┤ tenant-registry-new
│ pg-keycloak │ stage │ certifai-new │
│ infisical │ calls │ compliance-new │
│ pg-infisical │ prod │ pg-stage │
│ redis-infis │ KC + │ mongo-stage │
│ gitea │ Stalwart│ qdrant-stage │
└───────┬────────┘ └──────────────────┘
│ PRIVATE NETWORK 10.0.0.0/16
┌────────┴─────────┐
│ │
┌──────▼───────┐ ┌───────▼──────┐
│ vm-control │ │ vm-app-prod │
│ │ │ │
│ portal │ │ certifai │
│ tenant-reg │ │ mongodb │
│ orca-ctrl │ │ litellm │
│ erpnext │ │ backend-comp │
│ frappe-hd │ │ ai-sdk │
│ mariadb │ │ admin-comp │
│ redis-erp │ │ pg-app │
│ stalwart │ │ qdrant │
└──────────────┘ │ minio │
└──────────────┘
Orca-Proxy routing (vm-edge, by Host header):
auth.breakpilot.com → 127.0.0.1:8443 (Keycloak, local on vm-edge)
erp.breakpilot.com → vm-control:8000 (ERPNext) [allowlist: our IPs only]
git.breakpilot.com → vm-edge:3000 (Gitea, local) [allowlist: our IPs only]
mail.breakpilot.com → vm-control:587 (Stalwart submission) [allowlist: VM internal only]
ns1.breakpilot.com → 127.0.0.1:53 (PowerDNS, local)
*.breakpilot.com → vm-control:3000 (customer portal)
Orca-Proxy routing (stage, by Host header):
*.stage.breakpilot.com → 127.0.0.1:3000 (stage portal — all subdomains route here)
4. Storage and Volume Requirements
Block volumes (Ceph 3x replicated, €0.10/GiB/mo) mounted to each VM.
┌──────────────┬───────────────────────────────────────────┬─────────┬─────────────────────┐
│ VM │ Data stores │ +Block │ Growth profile │
├──────────────┼───────────────────────────────────────────┼─────────┼─────────────────────┤
│ vm-edge │ pg-keycloak + pg-infisical + Gitea repos │ +50 GB │ Slow │
│ vm-control │ MariaDB (ERPNext) + Stalwart mail spool │ +250 GB │ Medium │
│ vm-app-prod │ MongoDB + pg-app + Qdrant + MinIO │ +500 GB │ Fast (scales w/ N) │
│ stage │ pg-stage + mongo-stage + qdrant-stage │ +50 GB │ Resets per release │
└──────────────┴───────────────────────────────────────────┴─────────┴─────────────────────┘
Each VM's root disk: 50 GB ephemeral, included in flavor price.
Object storage (S3, €0.02/GiB/mo single-region or €0.0496/GiB/mo geo-redundant):
┌─────────────────────────────────┬─────────┬──────────────────────────┐
│ Bucket │ Size │ Purpose │
├─────────────────────────────────┼─────────┼──────────────────────────┤
│ s3://backups (geo-redundant) │ ~500 GB │ Database dumps │
│ s3://seed-data │ ~30 GB │ Demo tenant fixtures │
│ s3://exports │ ~50 GB │ GDPR/offboarding ZIPs │
│ s3://audit-archive │ ~20 GB │ Old audit log overflow │
└─────────────────────────────────┴─────────┴──────────────────────────┘
5. Backup Requirements
All backups ship to SysEleven Object Storage (S3-compatible, geo-redundant DUS2 ↔ HAM1 for production-critical data). Backup jobs run as Orca one-shot containers on cron. Infisical holds the S3 credentials.
┌───────────────────────┬──────────────────┬────────────┬────────────┬──────────────────────┐
│ Data store │ Method │ Frequency │ Retention │ Owner (who restores) │
├───────────────────────┼──────────────────┼────────────┼────────────┼──────────────────────┤
│ pg-keycloak (vm-edge) │ pg_dump → S3-geo │ Every 6h │ 14 days │ Infra Plane │
│ pg-infisical (vm-edge)│ pg_dump → S3-geo │ Daily │ 30 days │ Infra Plane │
│ Gitea (vm-edge) │ gitea dump → S3 │ Daily │ 30 days │ Infra Plane │
│ Keycloak realm export │ KC export → S3 │ Daily │ 14 days │ Identity Plane (owns)│
│ Infisical store │ encrypted → S3 │ Daily │ 30 days │ Infra Plane │
│ MariaDB (vm-control) │ mysqldump → S3 │ Every 6h │ 30 days │ Control Plane │
│ Stalwart queue/store │ tar → S3 │ Daily │ 7 days │ Control Plane │
│ pg-app (vm-app-prod) │ pg_dump → S3-geo │ Every 6h │ 30 days │ App Plane (owns RPO) │
│ MongoDB (vm-app-prod) │ mongodump → S3 │ Daily │ 30 days │ App Plane │
│ MinIO (vm-app-prod) │ mc mirror → S3 │ Daily │ 90 days │ App Plane │
│ Qdrant (vm-app-prod) │ API snap → S3 │ Daily │ 14 days │ App Plane (rebuild) │
│ stage * │ no backup │ — │ — │ — (ephemeral) │
│ Orca config (IaC) │ Gitea (VCS) │ On commit │ Forever │ Infra Plane │
└───────────────────────┴──────────────────┴────────────┴────────────┴──────────────────────┘
RPO by data criticality
CRITICAL (RPO ≤ 6h)
pg-keycloak — org memberships, IdP config
pg-app — tenant registry, compliance records
MariaDB/ERPNext — sales orders, invoices, contracts
IMPORTANT (RPO ≤ 24h)
MongoDB — chat history, user preferences
MinIO — compliance evidence documents
pg-infisical — encrypted secrets
Stalwart store — inbound webhooks, bounce records
RECOVERABLE (RPO ≤ 48h, rebuildable)
Qdrant — vector index (rebuildable from MinIO source documents)
Gitea — code (mirrored on dev machines)
Keycloak export — org structure (pg-keycloak is primary)
NOT BACKED UP
stage (any data) — by design; restored from seed bundles on each deploy
redis-* — caches; restart cold
6. Constraint Framework
Constraint types
AVAILABILITY — required uptime percentage over rolling 30 days
RTO — Recovery Time Objective: max time to restore service after failure
RPO — Recovery Point Objective: max acceptable data loss window
IaC — service must be declared in Orca config, no manual container runs in prod
SECRET_HYGIENE — all secrets via Infisical machine identity; no env files, no hardcoded values
NETWORK — whether service is internet-exposed or internal-only
DATA_RESIDENCY — all data must remain in EU (SysEleven DUS2 + HAM1)
AUDIT_TRAIL — all mutating actions logged (who, what, when, from where)
IMMUTABILITY — config changes go through Gitea → Orca pipeline, not manual SSH
STAGE_ISOLATION— stage tenant cannot mutate any prod data; reads-only against prod KC + TR
Plane ownership of constraints
Planes are the SLA + ownership abstraction; clusters are the operational unit. The plane that owns a constraint owns it regardless of which cluster hosts the service. Identity and Infra are co-tenant in the breakpilot-edge cluster on vm-edge; they remain distinct planes for purposes of SLA + ownership accounting.
╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ IDENTITY PLANE (cluster: breakpilot-edge, on vm-edge) ║
║ ║
║ Owns / defines: ║
║ AVAILABILITY — must be ≥ 99.5% (root dep for everything) ║
║ RTO — ≤ 15 min ║
║ AUDIT_TRAIL — realm-level audit (logins, token issuance, IdP events) ║
║ DATA_RESIDENCY— Keycloak realm data must stay EU ║
║ STAGE_ISOLATION— rate-limits stage_client_id; rejects stage JWTs in prod audiences ║
║ ║
║ Co-tenant note: shares vm-edge with Infra Plane services. JVM heap pinned to 1.5 GB ║
║ in Orca manifest (-Xmx1500m) so it cannot starve PowerDNS / Infisical. Escape hatch ║
║ if the heap fight shows up: flip placement.node to a new vm-edge-identity VM inside ║
║ the SAME breakpilot-edge cluster — no schema migration, just a container move. ║
║ ║
║ KC instance is REPURPOSED from the existing CERTifAI Keycloak at M2.1 (realm export ║
║ + user database move; hostname swap), not stood up fresh. ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ CONTROL PLANE (cluster: breakpilot-control, on vm-control) ║
║ ║
║ Owns / defines: ║
║ RPO (tenant) — tenant registry & compliance schemas RPO ≤ 6h ║
║ RPO (ERPNext) — sales orders, invoices RPO ≤ 6h ║
║ AUDIT_TRAIL — all portal actions (invites, IdP changes, impersonations) ║
║ AVAILABILITY — portal ≥ 99.5%; ERPNext ≥ 99% (internal) ║
║ RTO (portal) — ≤ 10 min ║
║ RTO (ERPNext) — ≤ 60 min ║
║ ║
║ Co-tenant note: ERPNext + Portal + Stalwart on one VM. Orca resource limits enforced: ║
║ portal: 1 GB memory cap ║
║ erpnext: 6 GB memory cap ║
║ mariadb: 3 GB memory cap ║
║ stalwart: 1 GB memory cap ║
║ tenant-registry: 500 MB ║
║ ║
║ Stage callers: vm-app-stage in breakpilot-app calls THIS tenant-registry under ║
║ tenant.kind = "stage" (read-only for stage tenants). No duplicated tenant-registry. ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ APP PLANE (was "Data Plane"; cluster: breakpilot-app, on vm-app-prod + vm-app-stage) ║
║ ║
║ Owns / defines: ║
║ DATA_RESIDENCY — all customer data (MongoDB, pg-app, MinIO) must stay EU ║
║ RPO (product) — compliance records ≤ 6h; chat history ≤ 24h ║
║ DATA_ISOLATION — every query scoped by org_id/tenant_id ║
║ AUDIT_TRAIL — product-level actions ║
║ AVAILABILITY — CERTifAI ≥ 99.5%; compliance ≥ 99.5% ║
║ ║
║ Co-tenant note: vm-app-prod is the SCALE driver. When it hits 80% RAM, bump flavor ║
║ (m2.medium → m2.large → m2.xlarge). See §13 Growth Trajectory. ║
║ ║
║ Stage VM (vm-app-stage) lives in the SAME cluster — same config, different physical ║
║ workload. Stage authenticates against the prod Identity plane (breakpilot-edge KC) ║
║ and reads tenant config from the prod Control plane (breakpilot-control ║
║ tenant-registry) — never duplicates identity or control. ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ INFRA PLANE (cluster: breakpilot-edge, on vm-edge, alongside Identity) ║
║ ║
║ Owns / enforces ALL of: ║
║ BACKUP — executes all backup jobs (pg_dump, mongodump, mc mirror) ║
║ IaC — ALL services declared in Orca config; no manual prod changes ║
║ IMMUTABILITY — config changes: Gitea commit → Gitea Actions → Orca API only ║
║ SECRET_HYGIENE— Infisical (on vm-edge); provisions machine identities ║
║ NETWORK — Orca-Proxy rules; VM firewall; no direct VM public exposure ║
║ DATA_RESIDENCY— VM region = SysEleven DUS2; backups geo-redundant DUS2↔HAM1 ║
║ AVAILABILITY — Orca restart policies, health checks ║
║ COLD_START — enforces startup ordering (see §10 Scenario F) ║
║ STAGE_ISOLATION— Infisical secret-path scoping for stage_app identity ║
║ ║
║ Per-cluster note: each of the three cluster repos has its own [backup] block + S3 ║
║ bucket. A runaway backup on breakpilot-app cannot fill breakpilot-edge's bucket. ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
7. SLA Table
┌───────────────────────┬──────────────┬─────────┬─────────┬────────────────────────────────┐
│ Service │ Availability │ RTO │ RPO │ Host VM │
├───────────────────────┼──────────────┼─────────┼─────────┼────────────────────────────────┤
│ Orca-Proxy │ 99.9% │ 5 min │ N/A │ vm-edge │
│ PowerDNS │ 99.9% │ 5 min │ N/A │ vm-edge │
│ Keycloak │ 99.5% │ 15 min │ 6h │ vm-edge (root auth dep) │
│ Infisical │ 99.5% │ 30 min │ 24h │ vm-edge (running svcs survive) │
│ Gitea │ 99% │ 2h │ 24h │ vm-edge (dev machines mirror) │
│ Customer Portal │ 99.5% │ 10 min │ N/A │ vm-control │
│ Tenant Registry │ 99.5% │ 10 min │ 6h │ vm-control │
│ ERPNext │ 99% │ 60 min │ 6h │ vm-control (internal only) │
│ Frappe HD │ 99% │ 60 min │ 24h │ vm-control │
│ MariaDB │ 99.5% │ 20 min │ 6h │ vm-control │
│ Stalwart Mail │ 99% │ 60 min │ 24h │ vm-control │
│ CERTifAI │ 99.5% │ 10 min │ 24h │ vm-app-prod │
│ MongoDB │ 99.5% │ 20 min │ 24h │ vm-app-prod │
│ LiteLLM │ 99% │ 5 min │ N/A │ vm-app-prod │
│ backend-compliance │ 99.5% │ 10 min │ 6h │ vm-app-prod │
│ ai-compliance-sdk │ 99.5% │ 10 min │ 6h │ vm-app-prod │
│ pg-app │ 99.9% │ 20 min │ 6h │ vm-app-prod (SPOF — RISK-1) │
│ MinIO │ 99.5% │ 30 min │ 24h │ vm-app-prod │
│ Qdrant │ 99% │ 2h │ 24h │ vm-app-prod (rebuildable) │
│ stage (any service) │ 95% │ best ef.│ N/A │ stage (ephemeral; no SLA) │
└───────────────────────┴──────────────┴─────────┴─────────┴────────────────────────────────┘
8. IaC Constraint (Orca)
Every production service declared in Orca config. No exceptions.
Rules
1. ALL containers run via Orca manifests committed to Gitea
→ /orca/manifests/{vm-name}/{service-name}.toml
→ Changes go through: Gitea PR → Gitea Actions lint → Orca API apply
2. NO manual docker run / docker-compose up on any production VM
→ SSH to prod VMs allowed for debugging only; no state changes
3. Secrets are NEVER in Orca manifests
→ Manifests reference Infisical paths, not values
→ Bootstrap exception: Keycloak DB URI in Orca env (Keycloak runs ON vm-edge alongside
Infisical, so chicken-and-egg is solved by Orca env file, not Infisical lookup)
4. Restart policy: always (Orca restarts crashed containers with exponential backoff)
→ Health check per service (HTTP /health or TCP probe)
5. Resource limits MANDATORY in every manifest
→ On a 3-VM prod, co-tenant noise is the single biggest risk; limits are non-negotiable
→ See §6 Plane ownership "Co-tenant note" boxes for the per-service caps
6. Orca controller state itself is recoverable
→ Manifest files in Gitea = desired state
→ Loss of Orca controller = re-apply manifests from Gitea, services continue running
7. Stage app gets its own Infisical scope
→ /stage/* path; no prod-DB credentials reach this scope
→ Enforced at Infisical machine-identity level, not in app code
Gitea Actions pipeline for infra changes
infra change committed to Gitea
│
├── lint: validate Orca manifest schema
├── diff: show what changes will be applied (orca plan)
├── (manual approval gate for vm-edge changes — touches auth root)
└── apply: POST to Orca Controller API → rolling update
9. Dependency Graph
Arrows = "requires to function." Dashed = soft (degrades, doesn't fail). Intra-VM dependencies elided for clarity (e.g. Keycloak ↔ pg-keycloak are on the same host and start together).
EXTERNAL
AI APIs
(OpenAI / Anthropic)
│
│ (soft)
▼
┌──────────────────────────────────────────────────────────────────┐
│ vm-edge (Identity + Infra) │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ pg-keycloak ──► keycloak │ │
│ │ pg-infisical ─► infisical ◄── (all VMs pull on startup) │ │
│ │ redis-infis ──► infisical │ │
│ │ (sqlite) ─────► gitea │ │
│ │ powerdns-auth (no deps) │ │
│ │ orca-proxy (route table only; backends are remote) │ │
│ └────────────────────────────────────────────────────────────┘ │
│ │ Keycloak JWKS │ Infisical /secrets │
│ │ │ │
└────────────────────────────┼────────────────┼────────────────────┘
▼ ▼
┌──────────────────────────────────────────────────────────────────┐
│ vm-control (Control) │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ mariadb + redis-erp ──► erpnext + frappe-hd │ │
│ │ (intra) ─────────────► stalwart │ │
│ │ ──────────────────────► customer-portal │ │
│ │ ──────────────────────► tenant-registry ──► pg-app (vm-app-prod)│ │
│ └────────────────────────────────────────────────────────────┘ │
│ │ tenant-registry API │
└────────────────────────────┼─────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────────────────────┐
│ vm-app-prod (Data) │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ mongodb ───► certifai ◄── (vm-edge JWKS, vm-edge secrets) │ │
│ │ litellm ───► certifai, ai-compliance-sdk │ │
│ │ pg-app ────► tenant-registry-on-vm-control, backend-compl,│ │
│ │ ai-compliance-sdk │ │
│ │ qdrant ────► ai-compliance-sdk │ │
│ │ minio ────► backend-compliance │ │
│ │ backend-compliance ──► admin-compliance │ │
│ └────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────┐
│ stage (App plane only) │
│ Calls vm-edge:8443 (KC) + vm-control:587 (Stalwart submission) │
│ Calls Polar SANDBOX (never prod Polar webhook URL) │
│ Its own ephemeral DBs; cannot read prod data │
└──────────────────────────────────────────────────────────────────┘
Simplified critical path (customer login → product use)
DNS (vm-edge PowerDNS)
│
▼
orca-proxy (vm-edge)
│
├──► keycloak (vm-edge) ──► pg-keycloak (intra-VM)
│
└──► customer-portal (vm-control)
├──► tenant-registry (vm-control) ──► pg-app (vm-app-prod)
├──► certifai (vm-app-prod) ──► mongodb (intra-VM)
└──► backend-compliance (vm-app-prod) ──► pg-app (intra-VM)
──► ai-sdk ──► qdrant + minio
──► litellm ──► [external AI APIs]
10. Failure Scenarios and Deadlock Analysis
Scenario A — vm-edge fails (HIGHEST SEVERITY)
Impact: TOTAL outage. Nothing reachable from internet.
No DNS. No TLS. No auth. No new logins. Running JWTs expire within 15 min,
then ALL services start returning 401.
Backstage and customer portal both fully blocked.
Stage also blocked (depends on prod Keycloak).
Cascade: T+0: DNS fails → orca-proxy unreachable
T+5m: existing JWTs still valid; portal cached → partial reads work
T+15m: JWTs expire → full outage
Deadlock: None — services downstream don't deadlock, they just fail closed
Recovery: 1. Spin up vm-edge-spare (cold standby, same Orca config) — ~3 min provision
2. Restore pg-keycloak + pg-infisical from latest backup — ~5 min
3. Swap registrar NS records to spare IP (TTL 60s) — ~2 min propagation
4. Restart all services on vm-edge-spare via Orca apply — ~3 min
Total RTO target: 15 min
Mitigation: COLD STANDBY vm-edge-spare. Same Orca config committed in Gitea.
Provision cost when idle: €0 (only billed when running).
Test recovery quarterly.
Severity: CRITICAL — single host owns 3 root dependencies (DNS, auth, secrets)
Cost of fix at Tier C: split vm-edge into vm-edge + vm-identity + vm-secrets
(back toward original 7-VM design) — €100/mo extra
Scenario B — vm-control fails (NEW — consequence of plane consolidation)
Impact: customer-portal: DOWN → /[slug]/* all return 503
tenant-registry: DOWN → Keycloak protocol-mapper for products claim breaks
→ users can log in but see "No active products"
ERPNext + Frappe HD: DOWN → we cannot create sales orders or read tickets
Stalwart: DOWN → no outbound emails (trial nudges, exports, ticket replies)
MariaDB: DOWN → ERPNext queries fail; backups paused
Products (CERTifAI, compliance): UNAFFECTED (on vm-app-prod, JWTs still validate)
Existing logged-in users: can use products directly via product subdomain
IF they bookmark it; portal home is 503.
Cascade: T+0: portal 503; new tenant onboarding blocked (registry down)
T+15m: existing JWTs missing refreshed products claim
T+1h: trial emails not sent → trial nudge cadence breaks
Deadlock: None
Recovery: Restart vm-control containers via Orca. If MariaDB corrupt: restore mysqldump.
RTO target: 10 min (portal) / 60 min (ERPNext)
Mitigation: Multiple services co-hosted = single failure hits many SLAs.
Resource limits in Orca prevent ERPNext OOM from killing portal.
Quarterly drill: deliberately stop portal, measure recovery.
Severity: HIGH — three services down at once, but products keep serving customers
Cost of fix at Tier B/C: split vm-control → vm-portal + vm-ops (ERPNext)
— €64/mo extra at m2.small
Scenario C — vm-app-prod fails
Impact: tenant-registry queries: FAIL (pg-app down) → portal returns 503 for tenant lookup
customer-portal: DEGRADED (login works, dashboard fails)
CERTifAI: COMPLETELY DOWN
backend-compliance + ai-sdk + admin: COMPLETELY DOWN
ERPNext + Stalwart: UNAFFECTED
Cascade: T+0: products down; portal degraded
T+15m: support tickets pile up
Note: prod is partial — users see error pages but ERPNext + auth still work
Recovery: Restart vm-app-prod containers. If pg-app corrupt: restore from pg_dump (RPO 6h).
RTO target: 20 min
Mitigation: This is the SCALE-event VM. RISK-1 below makes this the worst SPOF:
one pg-app instance owns tenant_registry + compliance schemas.
HIGH PRIORITY fix: split pg-app into separate clusters at Tier B/C transition.
Severity: HIGH — products down, business operations (ERPNext) still work so we can
contact customers
Scenario D — LiteLLM fails
Impact: CERTifAI: AI features fail (summarization, chat completion).
CERTifAI dashboard, sessions: UNAFFECTED.
compliance AI generation: FAILS (DSFA/TOM/VVT generation blocked).
Compliance CRUD: UNAFFECTED.
Cascade: Soft degradation only. Products show "AI features temporarily unavailable" banner.
Deadlock: None.
Recovery: Restart LiteLLM on vm-app-prod (stateless, ~30s).
Severity: MEDIUM — graceful degradation by design
Scenario E — vm-app-stage compromised or buggy
Impact: On stage itself: stage portal serves bad data; stage testers see errors.
On prod: NONE if isolation rules in §2 are intact.
Worst case if isolation breaks:
- Stage code tries to call prod pg-app → fails (no creds in /stage/* Infisical)
- Stage emits real email → blocked by Stalwart recipient filter
- Stage triggers Polar charge → goes to sandbox, no real money
Cascade: None to prod by design.
Recovery: Roll back stage to previous image via Orca. RTO target: 5 min.
Mitigation: The 5 enforcement rules in §2 are the load-bearing controls. Verify quarterly
via deliberate red-team: try to write to prod pg-app from stage and confirm 401.
Severity: LOW (in prod) / HIGH (on stage, but stage SLA is 95%)
Scenario F — Full Cold Start (Power Loss, All VMs Restart Simultaneously)
Three VMs boot at once. Services must start in dependency order or services
crash-loop until their deps are ready.
DEADLOCK RISK: vm-control services (portal, tenant-registry) start before vm-app-prod
services (pg-app, certifai, compliance). They'll crash-loop ~2-5min
with backoff retries.
Same for ERPNext on vm-control trying to reach Keycloak on vm-edge.
RESOLUTION: Orca enforces cross-VM startup ordering via health-check dependencies.
Bootstrap exception: Keycloak DB URI in Orca env on vm-edge (not from
Infisical — chicken-and-egg solved).
Required cold start sequence:
Phase 0 — Data roots on vm-app-prod (parallel):
pg-app, mongodb, qdrant, minio
Phase 0 — Data roots on vm-control (parallel):
mariadb, redis-erpnext
Phase 0 — Data roots on vm-edge (parallel):
pg-keycloak, pg-infisical, redis-infisical
Phase 1 — Secrets + DNS on vm-edge:
infisical (needs: pg-infisical, redis-infisical)
powerdns-auth (no deps)
Phase 2 — Identity on vm-edge:
keycloak (needs: pg-keycloak [Phase 0], infisical [Phase 1])
gitea (needs: sqlite; ready from Phase 0)
Phase 3 — Control on vm-control + Data services on vm-app-prod (parallel):
tenant-registry (needs: keycloak [Phase 2], pg-app [Phase 0, remote])
erpnext + frappe-hd (needs: mariadb, redis-erpnext [Phase 0], keycloak [Phase 2])
stalwart (needs: infisical [Phase 1])
litellm (needs: infisical)
certifai (needs: keycloak, mongodb, litellm)
backend-compliance (needs: keycloak, pg-app)
ai-compliance-sdk (needs: pg-app, qdrant, litellm)
admin-compliance (needs: backend + sdk)
Phase 4 — Customer-facing on vm-control:
customer-portal (needs: keycloak, tenant-registry)
Phase 5 — Gateway on vm-edge (last):
orca-proxy (waits for all backends healthy before opening listener)
Estimated cold-start time: 6-10 minutes (faster than 7-VM since less network roundtrip)
Scenario G — Tenant Registry fails
Impact: Portal cannot resolve tenant from subdomain → /[slug]/* all 503
Keycloak protocol mapper cannot get products claim → JWT missing field
→ users can log in but see "No active products"
Products (CERTifAI, compliance) themselves: UNAFFECTED if already authenticated
Cascade: New logins degraded.
Existing sessions continue.
Deadlock: None.
Recovery: Restart tenant-registry on vm-control. pg-app on vm-app-prod must be healthy.
RTO target: ≤ 60s
Mitigation: Portal caches slug → tenant mapping with 60s TTL.
Short outage invisible to customers.
Severity: MEDIUM
11. Cross-Dependency Summary Table
Needs → │PG-KC│PG-Inf│PG-App│Mongo│Maria│Redis│Minio│Qdrant│ KC │Infis│Lit. │T.Reg│
─────────────────────┼─────┼──────┼──────┼─────┼─────┼─────┼─────┼──────┼─────┼─────┼─────┼─────┤
keycloak │ ● │ │ │ │ │ │ │ │ │ ◐* │ │ │
infisical │ │ ● │ │ │ │ ● │ │ │ │ │ │ │
gitea │ │ │ │ │ │ │ │ │ │ ● │ │ │
tenant-registry │ │ │ ● │ │ │ │ │ │ ● │ ● │ │ │
customer-portal │ │ │ │ │ │ │ │ │ ● │ ● │ │ ● │
erpnext │ │ │ │ │ ● │ ● │ │ │ ● │ ● │ │ │
frappe-hd │ │ │ │ │ ● │ ● │ │ │ │ ● │ │ │
stalwart │ │ │ │ │ │ │ │ │ │ ● │ │ │
certifai │ │ │ │ ● │ │ │ │ │ ● │ ● │ ◐ │ │
litellm │ │ │ │ │ │ │ │ │ │ ● │ │ │
backend-compl. │ │ │ ● │ │ │ │ │ │ ● │ ● │ │ │
ai-compl-sdk │ │ │ ● │ │ │ │ │ ● │ │ ● │ ◐ │ │
admin-compl. │ │ │ │ │ │ │ │ │ │ │ │ │
orca-proxy │ │ │ │ │ │ │ │ │ │ │ │ │
stage-app │ │ │ │ │ │ │ │ │ ● │ ◑ │ │ ◑ │
● = hard dependency (cannot start without)
◐ = soft dependency (starts, features degrade)
◑ = stage-only read-mostly dependency (writes blocked by Infisical scope)
◐*= bootstrap exception (Keycloak DB URI in Orca env on vm-edge, not Infisical)
12. Open Infrastructure Risks (Priority Order)
RISK-1 pg-app (vm-app-prod) is a single instance serving tenant_registry + compliance schemas.
One crash blocks portal AND compliance product simultaneously.
→ Mitigation: split into pg-registry + pg-compliance at Tier B (200 customers).
Move pg-registry to its own DBaaS PostgreSQL cluster (€213/mo).
Priority: HIGH — fix before 100 customers; flagged also in COST_PLAN.md
RISK-2 vm-edge is a single VM owning 3 root dependencies (DNS, auth, secrets).
Failure = total external outage. Highest blast radius in the system.
→ Mitigation:
Phase A: cold-standby vm-edge-spare (idle cost €0; tested quarterly)
Phase B (Tier C, 500 cust): split vm-edge into vm-edge + vm-identity + vm-secrets
Priority: HIGH
RISK-3 vm-control hosts 5 service groups (portal, tenant-registry, ERPNext, Frappe HD,
Stalwart). Co-tenant noise risk; one OOM kills the others.
→ Mitigation:
Phase A: hard Orca resource limits per service (see §6 co-tenant notes)
Phase B (Tier B): split vm-control → vm-portal + vm-ops at €64/mo extra
Priority: MEDIUM
RISK-4 Keycloak is a single instance with no clustering.
Any Keycloak outage = total auth failure within JWT TTL.
→ Mitigation: short-term: tested runbook + 15min RTO target
long-term: Keycloak active-passive cluster (Phase 2, on split vm-identity)
Priority: MEDIUM
RISK-5 Stage isolation depends on 5 enforcement controls (see §2 table).
If any one breaks, stage code can affect prod customers.
→ Mitigation: quarterly red-team verification of each control.
Especially: Infisical secret-path scoping and Stalwart recipient filter.
Priority: MEDIUM — easy to forget once it's working
RISK-6 Infisical downtime during multi-VM restart causes delayed cold start.
→ Mitigation: Orca startup ordering + bootstrap secrets for Keycloak only
Priority: LOW — documented runbook; cold start is rare
RISK-7 ERPNext → Tenant Registry webhook has no guaranteed delivery.
Failed activation = tenant not active after contract signed.
→ Mitigation: Frappe retry + idempotent /activate endpoint + manual Backstage trigger
Priority: LOW
RISK-8 LiteLLM calls external AI APIs (OpenAI / Anthropic).
→ Mitigation: LiteLLM fallback routing; products degrade gracefully.
Priority: LOW — external dependency, by design
13. Growth Trajectory — when to add VMs
The locked 4-VM topology is right for 5–~200 customers. Past that, expect to add VMs back in this order:
Tier A (5–200 cust): 4 VMs as locked €192/mo compute (36M upfront)
↓
Tier B (200–500): Bump vm-app-prod m2.med → m2.large +€64/mo
Add cold-standby vm-edge-spare +€0 (idle, paid only on swap)
↓
Tier C (500–1000): Split vm-app-prod: vm-app-prod + vm-app-prod-db +€64/mo
(postgres-app moves to its own VM, or DBaaS cluster +€213/mo)
Split vm-control: vm-control + vm-ops +€64/mo
(ERPNext + MariaDB + Stalwart move to vm-ops)
↓
Tier D (1000–2000): Split vm-edge: vm-edge + vm-identity + vm-secrets +€96/mo
HA Keycloak active-passive on 2× vm-identity +€32/mo
Octavia Load Balancer Double Instance +€58/mo
vm-app-prod m2.large → m2.xlarge or 2× +€128–256/mo
↓
Final topology ≈ 8 prod VMs + DBaaS
Each step is justified by a measurable signal (>80% RAM, >70% CPU, sustained queue depth, or a specific outage scenario). Never split preemptively.
14. Cost summary (see COST_PLAN.md for full breakdown)
| Mode | Compute €/mo | Storage €/mo | Network €/mo | Total net | + 19% VAT |
|---|---|---|---|---|---|
| On-Demand | 434.50 | 112 | 2.92 | 549.42 | 653.81 |
| 12-month commit | 295.20 | 112 | 2.92 | 410.12 | 488.04 |
| 36-month no upfront | 216.00 | 112 | 2.92 | 330.92 | 393.79 |
| 36-month upfront | 192.00 | 112 | 2.92 | 306.92 | 365.23 |
Plus €6,912 net one-time payment if signing 36M-upfront for the compute portion.
End of document. Review quarterly or after any significant infrastructure change. Topology last locked 2026-05-18.