platform/docs
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(arch): align INFRASTRUCTURE + PLATFORM_ARCHITECTURE to cluster split (#7)
ci / shared (push) Successful in 11s
Details

Browse Source 
Same 4 VMs / flavors / SLA. 3 Orca clusters (was 1), 1 cluster per plane: breakpilot-edge (Identity + Infra), breakpilot-control (Control), breakpilot-app (App, prod + stage VMs). vm-data → vm-app-prod; stage VM → vm-app-stage. Data plane → App plane. KC co-tenant on vm-edge with JVM heap pinned. Companions: PR #6 here (impl plan) + orca-platform PR #6 (cluster manifests).
This commit was merged in pull request #7.
This commit is contained in:
sharang
2026-06-30 21:30:47 +00:00
parent a8593091c2
commit 1fd03592c0
2 changed files with 119 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
INFRASTRUCTURE.md
+85 -64
View File
@@ -1,55 +1,59 @@
# Infrastructure Specification
**Status:** Locked Topology
**Authors:** Sharang, Benjamin
**Date:** 2026-05-11 (topology lock: 2026-05-18)
**Date:** 2026-05-11 (topology lock: 2026-05-18; cluster split: 2026-06-30)
**Companion docs:** PLATFORM_ARCHITECTURE.md, IMPLEMENTATION_PLAN.md, COST_PLAN.md
**Cloud provider:** SysEleven Cloud Services (DUS2, OpenStack)
**Cloud provider:** SysEleven Cloud Services (DUS2, OpenStack) — multi-VM rollout gated on legal entity
---
> **2026-06-30 cluster-split decision.** The 4 VMs from the May 18 lock are unchanged in count and flavor. What changes is how they're organized: **one Orca cluster per plane**, three cluster repos (`platform/breakpilot-edge`, `platform/breakpilot-control`, `platform/breakpilot-app`). Identity and Infra are co-tenant on `vm-edge` inside `breakpilot-edge` (single-VM core; KC JVM heap pinned so it cannot starve PowerDNS / Infisical). The "Data plane" is renamed to **App plane** to reflect that it carries product workloads, not just data stores. `vm-data` is renamed `vm-app-prod`, and the standalone `stage` VM is renamed `vm-app-stage` — both live in the same `breakpilot-app` cluster. Multi-VM rollout is gated on the legal entity being established so we can sign SysEleven business contracts; until then, single-VM ops continues via `~/workspace/orca-infra`. See `IMPLEMENTATION_PLAN.md §1.1 / M1.2` and `platform/orca-platform/clusters/README.md` for the migration mechanics. **The SLA targets in §6 / §7 below are unaffected** — they apply at the plane level regardless of which cluster a plane lives in.
## 1. VM Inventory
**Four billable VMs total.** Three in production (one per plane after collapsing Identity+Infra), one in stage. Dev runs entirely on developer laptops via docker-compose.
**Four billable VMs across three Orca clusters.** Dev runs entirely on developer laptops via docker-compose.
```
┌──────────────┬─────────────────┬────────────────────────┬───────────┬─────────────────┐
│ Name │ Env │ SysEleven flavor │ Public IP │ Planes owned │
├──────────────┼─────────────────┼────────────────────────┼───────────┼─────────────────┤
│ vm-edge │ prod │ m2.small (2v / 8 GB) │ YES (1) │ Identity + Infra│
│ vm-control │ prod │ m2.medium (4v / 16 GB) │ No │ Control │
│ vm-data │ prod │ m2.medium (4v / 16 GB) │ No │ Data │
│ stage │ stage │ m2.small (2v / 8 GB) │ YES (1) │ App plane only │
│ (dev) │ dev │ local docker-compose │ n/a │ all (in-memory) │
└──────────────┴─────────────────┴────────────────────────┴───────────┴─────────────────┘
┌──────────────────┬────────────────────┬──────────┬────────────────────────┬───────────┬─────────────────┐
│ Name │ Cluster │ Env │ SysEleven flavor │ Public IP │ Planes owned │
├──────────────────┼────────────────────┼──────────┼────────────────────────┼───────────┼─────────────────┤
│ vm-edge │ breakpilot-edge │ prod │ m2.small (2v / 8 GB) │ YES (1) │ Identity + Infra│
│ vm-control │ breakpilot-control │ prod │ m2.medium (4v / 16 GB) │ No │ Control │
│ vm-app-prod │ breakpilot-app │ prod │ m2.medium (4v / 16 GB) │ No │ App (prod) │
│ vm-app-stage │ breakpilot-app │ stage │ m2.small (2v / 8 GB) │ YES (1) │ App (stage) │
│ (dev) │ — │ dev │ local docker-compose │ n/a │ all (in-memory) │
└──────────────────┴────────────────────┴──────────┴────────────────────────┴───────────┴─────────────────┘
```
**Total compute:** 48 GiB-RAM, 12 vCPU. **Monthly compute net: €192 (36M upfront) / €295 (12M) / €435 (On-Demand).** See COST_PLAN.md for the full three-mode table.
**Total compute:** 48 GiB-RAM, 12 vCPU. **Monthly compute net: €192 (36M upfront) / €295 (12M) / €435 (On-Demand).** See COST_PLAN.md for the full three-mode table. The cluster split does not change the per-VM bill (same flavors); the additional cost is operational (3 cluster repos to keep CI green in) and is offset by the independent failure-domain wins.
### Why this topology and not the previous 7-VM layout
The earlier draft proposed one VM per service group (vm-gateway, vm-identity, vm-secrets, vm-ops, vm-control, vm-certifai, vm-compliance). That gave maximum failure isolation but cost 132 GiB-RAM stage+prod. At 5 customers the isolation is unused — every VM ran at <10% utilisation. The locked topology buys back failure isolation incrementally as load grows (see §13 Growth Trajectory).
Critical isolations preserved even at 4 VMs:
- **vm-edge isolates identity from app workloads.** Keycloak JVM has its own page cache; ERPNext background jobs cannot starve token issuance.
- **vm-data isolates databases from stateless services.** All data-plane DBs share one host, but they're walled off from the portal + ERPNext + Stalwart competing on vm-control.
- **stage runs the app plane only.** It calls prod Keycloak + prod Tenant Registry under `tenant.kind = stage` rather than mirroring those services.
- **vm-edge isolates identity from app workloads.** Keycloak JVM has its own page cache + heap pin (`-Xmx1500m`) so ERPNext or product background jobs on a different VM cannot starve token issuance — and PowerDNS / Infisical on the same VM cannot be starved by KC either.
- **vm-app-prod isolates product databases from stateless services.** All app-plane DBs share one host, but they're walled off from the portal + ERPNext + Stalwart competing on vm-control.
- **vm-app-stage runs the app plane only.** It calls prod Keycloak (`breakpilot-edge`) + prod Tenant Registry (`breakpilot-control`) under `tenant.kind = stage` rather than mirroring those services. Stage and prod live in the **same** `breakpilot-app` Orca cluster on **different** VMs — same config, different physical workloads.
---
## 2. Service-to-VM Mapping
Each VM is owned by one Orca cluster (see §1). Service manifests live in that cluster's repo at `platform/breakpilot-<name>/services/...`.
```
vm-edge (prod, m2.small 8 GB, public IP)
vm-edge (cluster: breakpilot-edge, prod, m2.small 8 GB, public IP)
├── orca-proxy (Orca-managed; wildcard TLS terminator)
├── powerdns-auth (Orca-managed; authoritative DNS for breakpilot.com)
├── keycloak-26 (Orca-managed; JVM, ~1.5 GB heap)
├── keycloak-26 (Orca-managed; JVM, ~1.5 GB heap; repurposed from CERTifAI KC at M2.1)
├── postgres-keycloak (Orca-managed; dedicated PG instance for Keycloak only)
├── infisical (Orca-managed)
├── postgres-infisical (Orca-managed; dedicated PG instance for Infisical only)
├── redis-infisical (Orca-managed; ephemeral)
└── gitea (Orca-managed; SQLite backend to avoid a third PG)
vm-control (prod, m2.medium 16 GB)
vm-control (cluster: breakpilot-control, prod, m2.medium 16 GB)
├── customer-portal (Orca-managed; Next.js)
├── tenant-registry (Orca-managed; Go)
├── orca-controller (Orca core process; NOT a managed container)
@@ -59,7 +63,7 @@ vm-control (prod, m2.medium 16 GB)
├── redis-erpnext (Orca-managed)
└── stalwart-mail (Orca-managed; SMTP/IMAP/JMAP on mail.breakpilot.com)
vm-data (prod, m2.medium 16 GB)
vm-app-prod (cluster: breakpilot-app, prod, m2.medium 16 GB)
├── certifai-dashboard (Orca-managed)
├── mongodb (Orca-managed)
├── litellm (Orca-managed)
@@ -70,22 +74,23 @@ vm-data (prod, m2.medium 16 GB)
├── qdrant (Orca-managed)
└── minio (Orca-managed)
stage (stage, m2.small 8 GB, public IP)
vm-app-stage (cluster: breakpilot-app — SAME cluster as vm-app-prod, stage, m2.small 8 GB, public IP)
├── orca-proxy (light; only routes to stage app)
├── customer-portal (NEW VERSION under test)
├── tenant-registry (NEW VERSION under test, talks to ephemeral PG below)
├── tenant-registry (placeholder; stage actually calls PROD tenant-registry — manifest exists for parity, see §5 isolation rules)
├── certifai-dashboard (NEW VERSION under test)
├── backend-compliance (NEW VERSION under test)
├── ai-compliance-sdk (NEW VERSION under test)
├── admin-compliance (NEW VERSION under test)
├── litellm (light; same image as prod)
├── postgres-app-stage (ephemeral; lives entirely on stage VM)
├── postgres-app-stage (ephemeral; lives entirely on vm-app-stage)
├── mongodb-stage (ephemeral)
└── qdrant-stage (ephemeral, tiny corpus)
Calls OUT to prod:
→ auth.breakpilot.com (Keycloak token issuance, under stage client_id)
→ mail.breakpilot.com (Stalwart SMTP, recipient filter forces +stage@ only)
→ auth.breakpilot.com (PROD Keycloak in breakpilot-edge, under stage client_id, tenant.kind = "stage")
→ registry.breakpilot.com (PROD tenant-registry in breakpilot-control, read-only for stage tenants)
→ mail.breakpilot.com (PROD Stalwart in breakpilot-control, recipient filter forces +stage@ only)
→ Polar SANDBOX webhook URL (NEVER prod Polar)
→ no calls to prod Postgres-app, MariaDB, MongoDB
```
@@ -129,7 +134,7 @@ stage (stage, m2.small 8 GB, public IP)
┌────────┴─────────┐
│ │
┌──────▼───────┐ ┌───────▼──────┐
│ vm-control │ │ vm-data │
│ vm-control │ │ vm-app-prod │
│ │ │ │
│ portal │ │ certifai │
│ tenant-reg │ │ mongodb │
@@ -166,7 +171,7 @@ Block volumes (Ceph 3x replicated, €0.10/GiB/mo) mounted to each VM.
├──────────────┼───────────────────────────────────────────┼─────────┼─────────────────────┤
│ vm-edge │ pg-keycloak + pg-infisical + Gitea repos │ +50 GB │ Slow │
│ vm-control │ MariaDB (ERPNext) + Stalwart mail spool │ +250 GB │ Medium │
│ vm-data │ MongoDB + pg-app + Qdrant + MinIO │ +500 GB │ Fast (scales w/ N) │
│ vm-app-prod │ MongoDB + pg-app + Qdrant + MinIO │ +500 GB │ Fast (scales w/ N) │
│ stage │ pg-stage + mongo-stage + qdrant-stage │ +50 GB │ Resets per release │
└──────────────┴───────────────────────────────────────────┴─────────┴─────────────────────┘
@@ -200,10 +205,10 @@ All backups ship to **SysEleven Object Storage** (S3-compatible, geo-redundant D
│ Infisical store │ encrypted → S3 │ Daily │ 30 days │ Infra Plane │
│ MariaDB (vm-control) │ mysqldump → S3 │ Every 6h │ 30 days │ Control Plane │
│ Stalwart queue/store │ tar → S3 │ Daily │ 7 days │ Control Plane │
│ pg-app (vm-data) │ pg_dump → S3-geo │ Every 6h │ 30 days │ Data Plane (owns RPO)│
│ MongoDB (vm-data) │ mongodump → S3 │ Daily │ 30 days │ Data Plane │
│ MinIO (vm-data) │ mc mirror → S3 │ Daily │ 90 days │ Data Plane │
│ Qdrant (vm-data) │ API snap → S3 │ Daily │ 14 days │ Data Plane (rebuild) │
│ pg-app (vm-app-prod) │ pg_dump → S3-geo │ Every 6h │ 30 days │ App Plane (owns RPO) │
│ MongoDB (vm-app-prod) │ mongodump → S3 │ Daily │ 30 days │ App Plane │
│ MinIO (vm-app-prod) │ mc mirror → S3 │ Daily │ 90 days │ App Plane │
│ Qdrant (vm-app-prod) │ API snap → S3 │ Daily │ 14 days │ App Plane (rebuild) │
│ stage * │ no backup │ — │ — │ — (ephemeral) │
│ Orca config (IaC) │ Gitea (VCS) │ On commit │ Forever │ Infra Plane │
└───────────────────────┴──────────────────┴────────────┴────────────┴──────────────────────┘
@@ -254,11 +259,11 @@ STAGE_ISOLATION— stage tenant cannot mutate any prod data; reads-only against
### Plane ownership of constraints
Even though planes now share VMs, the **ownership model is unchanged** — the plane that owns a constraint owns it regardless of which VM hosts the service. The Infra Plane (now collapsed onto vm-edge alongside the Identity plane) still mechanically enforces backup, IaC, secrets, and network constraints.
Planes are the **SLA + ownership abstraction**; clusters are the **operational unit**. The plane that owns a constraint owns it regardless of which cluster hosts the service. Identity and Infra are co-tenant in the `breakpilot-edge` cluster on `vm-edge`; they remain distinct planes for purposes of SLA + ownership accounting.
```
╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ IDENTITY PLANE (on vm-edge) ║
║ IDENTITY PLANE (cluster: breakpilot-edge, on vm-edge) ║
║ ║
║ Owns / defines: ║
║ AVAILABILITY — must be ≥ 99.5% (root dep for everything) ║
@@ -268,11 +273,16 @@ Even though planes now share VMs, the **ownership model is unchanged** — the p
║ STAGE_ISOLATION— rate-limits stage_client_id; rejects stage JWTs in prod audiences ║
║ ║
║ Co-tenant note: shares vm-edge with Infra Plane services. JVM heap pinned to 1.5 GB ║
║ in Orca manifest so it cannot starve PowerDNS / Infisical. ║
║ in Orca manifest (-Xmx1500m) so it cannot starve PowerDNS / Infisical. Escape hatch ║
║ if the heap fight shows up: flip placement.node to a new vm-edge-identity VM inside ║
║ the SAME breakpilot-edge cluster — no schema migration, just a container move. ║
║ ║
║ KC instance is REPURPOSED from the existing CERTifAI Keycloak at M2.1 (realm export ║
║ + user database move; hostname swap), not stood up fresh. ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ CONTROL PLANE (on vm-control) ║
║ CONTROL PLANE (cluster: breakpilot-control, on vm-control) ║
║ ║
║ Owns / defines: ║
║ RPO (tenant) — tenant registry & compliance schemas RPO ≤ 6h ║
@@ -288,10 +298,13 @@ Even though planes now share VMs, the **ownership model is unchanged** — the p
║ mariadb: 3 GB memory cap ║
║ stalwart: 1 GB memory cap ║
║ tenant-registry: 500 MB ║
║ ║
║ Stage callers: vm-app-stage in breakpilot-app calls THIS tenant-registry under ║
║ tenant.kind = "stage" (read-only for stage tenants). No duplicated tenant-registry. ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ DATA PLANE (on vm-data) ║
║ APP PLANE (was "Data Plane"; cluster: breakpilot-app, on vm-app-prod + vm-app-stage) ║
║ ║
║ Owns / defines: ║
║ DATA_RESIDENCY — all customer data (MongoDB, pg-app, MinIO) must stay EU ║
@@ -300,12 +313,17 @@ Even though planes now share VMs, the **ownership model is unchanged** — the p
║ AUDIT_TRAIL — product-level actions ║
║ AVAILABILITY — CERTifAI ≥ 99.5%; compliance ≥ 99.5% ║
║ ║
║ Co-tenant note: this VM is the SCALE driver. When vm-data hits 80% RAM, bump flavor ║
║ Co-tenant note: vm-app-prod is the SCALE driver. When it hits 80% RAM, bump flavor ║
║ (m2.medium → m2.large → m2.xlarge). See §13 Growth Trajectory. ║
║ ║
║ Stage VM (vm-app-stage) lives in the SAME cluster — same config, different physical ║
║ workload. Stage authenticates against the prod Identity plane (breakpilot-edge KC) ║
║ and reads tenant config from the prod Control plane (breakpilot-control ║
║ tenant-registry) — never duplicates identity or control. ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ INFRA PLANE (on vm-edge, alongside Identity) ║
║ INFRA PLANE (cluster: breakpilot-edge, on vm-edge, alongside Identity) ║
║ ║
║ Owns / enforces ALL of: ║
║ BACKUP — executes all backup jobs (pg_dump, mongodump, mc mirror) ║
@@ -317,6 +335,9 @@ Even though planes now share VMs, the **ownership model is unchanged** — the p
║ AVAILABILITY — Orca restart policies, health checks ║
║ COLD_START — enforces startup ordering (see §10 Scenario F) ║
║ STAGE_ISOLATION— Infisical secret-path scoping for stage_app identity ║
║ ║
║ Per-cluster note: each of the three cluster repos has its own [backup] block + S3 ║
║ bucket. A runaway backup on breakpilot-app cannot fill breakpilot-edge's bucket. ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
```
@@ -339,14 +360,14 @@ Even though planes now share VMs, the **ownership model is unchanged** — the p
│ Frappe HD │ 99% │ 60 min │ 24h │ vm-control │
│ MariaDB │ 99.5% │ 20 min │ 6h │ vm-control │
│ Stalwart Mail │ 99% │ 60 min │ 24h │ vm-control │
│ CERTifAI │ 99.5% │ 10 min │ 24h │ vm-data │
│ MongoDB │ 99.5% │ 20 min │ 24h │ vm-data │
│ LiteLLM │ 99% │ 5 min │ N/A │ vm-data │
│ backend-compliance │ 99.5% │ 10 min │ 6h │ vm-data │
│ ai-compliance-sdk │ 99.5% │ 10 min │ 6h │ vm-data │
│ pg-app │ 99.9% │ 20 min │ 6h │ vm-data (SPOF — RISK-1) │
│ MinIO │ 99.5% │ 30 min │ 24h │ vm-data │
│ Qdrant │ 99% │ 2h │ 24h │ vm-data (rebuildable) │
│ CERTifAI │ 99.5% │ 10 min │ 24h │ vm-app-prod │
│ MongoDB │ 99.5% │ 20 min │ 24h │ vm-app-prod │
│ LiteLLM │ 99% │ 5 min │ N/A │ vm-app-prod │
│ backend-compliance │ 99.5% │ 10 min │ 6h │ vm-app-prod │
│ ai-compliance-sdk │ 99.5% │ 10 min │ 6h │ vm-app-prod │
│ pg-app │ 99.9% │ 20 min │ 6h │ vm-app-prod (SPOF — RISK-1) │
│ MinIO │ 99.5% │ 30 min │ 24h │ vm-app-prod │
│ Qdrant │ 99% │ 2h │ 24h │ vm-app-prod (rebuildable) │
│ stage (any service) │ 95% │ best ef.│ N/A │ stage (ephemeral; no SLA) │
└───────────────────────┴──────────────┴─────────┴─────────┴────────────────────────────────┘
```
@@ -433,13 +454,13 @@ Arrows = "requires to function." Dashed = soft (degrades, doesn't fail).
│ │ mariadb + redis-erp ──► erpnext + frappe-hd │ │
│ │ (intra) ─────────────► stalwart │ │
│ │ ──────────────────────► customer-portal │ │
│ │ ──────────────────────► tenant-registry ──► pg-app (vm-data)│ │
│ │ ──────────────────────► tenant-registry ──► pg-app (vm-app-prod)│ │
│ └────────────────────────────────────────────────────────────┘ │
│ │ tenant-registry API │
└────────────────────────────┼─────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────────────────────┐
│ vm-data (Data) │
│ vm-app-prod (Data) │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ mongodb ───► certifai ◄── (vm-edge JWKS, vm-edge secrets) │ │
│ │ litellm ───► certifai, ai-compliance-sdk │ │
@@ -470,9 +491,9 @@ Arrows = "requires to function." Dashed = soft (degrades, doesn't fail).
├──► keycloak (vm-edge) ──► pg-keycloak (intra-VM)
│
└──► customer-portal (vm-control)
├──► tenant-registry (vm-control) ──► pg-app (vm-data)
├──► certifai (vm-data) ──► mongodb (intra-VM)
└──► backend-compliance (vm-data) ──► pg-app (intra-VM)
├──► tenant-registry (vm-control) ──► pg-app (vm-app-prod)
├──► certifai (vm-app-prod) ──► mongodb (intra-VM)
└──► backend-compliance (vm-app-prod) ──► pg-app (intra-VM)
──► ai-sdk ──► qdrant + minio
──► litellm ──► [external AI APIs]
```
@@ -515,7 +536,7 @@ Impact: customer-portal: DOWN → /[slug]/* all return 503
ERPNext + Frappe HD: DOWN → we cannot create sales orders or read tickets
Stalwart: DOWN → no outbound emails (trial nudges, exports, ticket replies)
MariaDB: DOWN → ERPNext queries fail; backups paused
Products (CERTifAI, compliance): UNAFFECTED (on vm-data, JWTs still validate)
Products (CERTifAI, compliance): UNAFFECTED (on vm-app-prod, JWTs still validate)
Existing logged-in users: can use products directly via product subdomain
IF they bookmark it; portal home is 503.
Cascade: T+0: portal 503; new tenant onboarding blocked (registry down)
@@ -532,7 +553,7 @@ Cost of fix at Tier B/C: split vm-control → vm-portal + vm-ops (ERPNext)
— €64/mo extra at m2.small
```
### Scenario C — vm-data fails
### Scenario C — vm-app-prod fails
```
Impact: tenant-registry queries: FAIL (pg-app down) → portal returns 503 for tenant lookup
@@ -543,7 +564,7 @@ Impact: tenant-registry queries: FAIL (pg-app down) → portal returns 503 fo
Cascade: T+0: products down; portal degraded
T+15m: support tickets pile up
Note: prod is partial — users see error pages but ERPNext + auth still work
Recovery: Restart vm-data containers. If pg-app corrupt: restore from pg_dump (RPO 6h).
Recovery: Restart vm-app-prod containers. If pg-app corrupt: restore from pg_dump (RPO 6h).
RTO target: 20 min
Mitigation: This is the SCALE-event VM. RISK-1 below makes this the worst SPOF:
one pg-app instance owns tenant_registry + compliance schemas.
@@ -561,11 +582,11 @@ Impact: CERTifAI: AI features fail (summarization, chat completion).
Compliance CRUD: UNAFFECTED.
Cascade: Soft degradation only. Products show "AI features temporarily unavailable" banner.
Deadlock: None.
Recovery: Restart LiteLLM on vm-data (stateless, ~30s).
Recovery: Restart LiteLLM on vm-app-prod (stateless, ~30s).
Severity: MEDIUM — graceful degradation by design
```
### Scenario E — Stage VM compromised or buggy
### Scenario E — vm-app-stage compromised or buggy
```
Impact: On stage itself: stage portal serves bad data; stage testers see errors.
@@ -587,7 +608,7 @@ Severity: LOW (in prod) / HIGH (on stage, but stage SLA is 95%)
Three VMs boot at once. Services must start in dependency order or services
crash-loop until their deps are ready.
DEADLOCK RISK: vm-control services (portal, tenant-registry) start before vm-data
DEADLOCK RISK: vm-control services (portal, tenant-registry) start before vm-app-prod
services (pg-app, certifai, compliance). They'll crash-loop ~2-5min
with backoff retries.
Same for ERPNext on vm-control trying to reach Keycloak on vm-edge.
@@ -598,7 +619,7 @@ RESOLUTION: Orca enforces cross-VM startup ordering via health-check dependencie
Required cold start sequence:
Phase 0 — Data roots on vm-data (parallel):
Phase 0 — Data roots on vm-app-prod (parallel):
pg-app, mongodb, qdrant, minio
Phase 0 — Data roots on vm-control (parallel):
mariadb, redis-erpnext
@@ -613,7 +634,7 @@ Required cold start sequence:
keycloak (needs: pg-keycloak [Phase 0], infisical [Phase 1])
gitea (needs: sqlite; ready from Phase 0)
Phase 3 — Control on vm-control + Data services on vm-data (parallel):
Phase 3 — Control on vm-control + Data services on vm-app-prod (parallel):
tenant-registry (needs: keycloak [Phase 2], pg-app [Phase 0, remote])
erpnext + frappe-hd (needs: mariadb, redis-erpnext [Phase 0], keycloak [Phase 2])
stalwart (needs: infisical [Phase 1])
@@ -642,7 +663,7 @@ Impact: Portal cannot resolve tenant from subdomain → /[slug]/* all 503
Cascade: New logins degraded.
Existing sessions continue.
Deadlock: None.
Recovery: Restart tenant-registry on vm-control. pg-app on vm-data must be healthy.
Recovery: Restart tenant-registry on vm-control. pg-app on vm-app-prod must be healthy.
RTO target: ≤ 60s
Mitigation: Portal caches slug → tenant mapping with 60s TTL.
Short outage invisible to customers.
@@ -683,7 +704,7 @@ stage-app │ │ │ │ │ │ │
## 12. Open Infrastructure Risks (Priority Order)
```
RISK-1 pg-app (vm-data) is a single instance serving tenant_registry + compliance schemas.
RISK-1 pg-app (vm-app-prod) is a single instance serving tenant_registry + compliance schemas.
One crash blocks portal AND compliance product simultaneously.
→ Mitigation: split into pg-registry + pg-compliance at Tier B (200 customers).
Move pg-registry to its own DBaaS PostgreSQL cluster (€213/mo).
@@ -738,10 +759,10 @@ The locked 4-VM topology is right for 5–~200 customers. Past that, expect to a
```
Tier A (5–200 cust): 4 VMs as locked €192/mo compute (36M upfront)
↓
Tier B (200–500): Bump vm-data m2.med → m2.large +€64/mo
Tier B (200–500): Bump vm-app-prod m2.med → m2.large +€64/mo
Add cold-standby vm-edge-spare +€0 (idle, paid only on swap)
↓
Tier C (500–1000): Split vm-data: vm-data + vm-data-db +€64/mo
Tier C (500–1000): Split vm-app-prod: vm-app-prod + vm-app-prod-db +€64/mo
(postgres-app moves to its own VM, or DBaaS cluster +€213/mo)
Split vm-control: vm-control + vm-ops +€64/mo
(ERPNext + MariaDB + Stalwart move to vm-ops)
@@ -749,7 +770,7 @@ Tier C (500–1000): Split vm-data: vm-data + vm-data-db +€64/mo
Tier D (1000–2000): Split vm-edge: vm-edge + vm-identity + vm-secrets +€96/mo
HA Keycloak active-passive on 2× vm-identity +€32/mo
Octavia Load Balancer Double Instance +€58/mo
vm-data m2.large → m2.xlarge or 2× +€128–256/mo
vm-app-prod m2.large → m2.xlarge or 2× +€128–256/mo
↓
Final topology ≈ 8 prod VMs + DBaaS
```
PLATFORM_ARCHITECTURE.md
+31 -17
View File
@@ -1,10 +1,12 @@
# Platform Architecture — B2B Customer Portal
**Status:** Design Draft
**Authors:** Sharang, Benjamin
**Date:** 2026-05-11
**Date:** 2026-05-11 (cluster split: 2026-06-30)
---
> **2026-06-30 cluster-split decision.** The four-plane vocabulary in this document still applies as the **SLA + ownership abstraction**. What changed is the operational shape: planes are now hosted by **three Orca clusters** (one cluster per plane, with Identity and Infra co-tenant in the same cluster on a single VM). The "Data plane" is renamed **App plane** to reflect that it carries product workloads, not just data stores. Cluster repos: `platform/breakpilot-edge` (Identity + Infra), `platform/breakpilot-control` (Control), `platform/breakpilot-app` (App, with prod + stage VMs). Multi-VM rollout is gated on the legal entity being established. See `INFRASTRUCTURE.md §1`, `IMPLEMENTATION_PLAN.md §1.1 / M1.2`, and `platform/orca-platform/clusters/README.md` for the operational details.
## 1. Vision
We sell CERTifAI and breakpilot-compliance as modular B2B building blocks. Customers buy one or both and operate them inside a unified customer portal — without needing to understand that they are separate products under the hood.
@@ -28,24 +30,32 @@ Out of scope: breakpilot-dataroom, breakpilot-lehrer, breakpilot-pitch-deck.
## 3. The Four Planes
Four logical planes — but only **three Orca clusters**. Identity and Infra are co-tenant in the `breakpilot-edge` cluster on `vm-edge` per the 2026-06-30 cluster-split decision. The plane abstraction owns SLA and audit responsibilities; the cluster abstraction owns operational concerns (failure domain, backup policy, CI scope, IaC repo).
```
╔══════════════════════════════════════════════════════════════════╗
║ PLANE 1 — IDENTITY (logical root, all auth flows through here) ║
╚══════════════════════════════════════════════════════════════════╝
╔══════════════════════════════════════════════════════════════════════════════╗
║ PLANE 1 — IDENTITY cluster: breakpilot-edge VM: vm-edge ║
║ (logical root, all auth flows through here) ║
╚══════════════════════════════════════════════════════════════════════════════╝
↓ JWT
╔══════════════════════════════════════════════════════════════════╗
║ PLANE 2 — CONTROL (portal + ERPNext + tenant registry) ║
╚══════════════════════════════════════════════════════════════════╝
╔══════════════════════════════════════════════════════════════════════════════╗
║ PLANE 2 — CONTROL cluster: breakpilot-control VM: vm-control ║
║ (portal + ERPNext + tenant registry) ║
╚══════════════════════════════════════════════════════════════════════════════╝
↓ tenant-scoped API calls
╔══════════════════════════════════════════════════════════════════╗
║ PLANE 3 — DATA (CERTifAI + breakpilot-compliance) ║
╚══════════════════════════════════════════════════════════════════╝
╔══════════════════════════════════════════════════════════════════════════════╗
║ PLANE 3 — APP (was DATA) cluster: breakpilot-app VMs: vm-app-prod ║
║ (CERTifAI + breakpilot-compliance + data stores) + vm-app-stage║
╚══════════════════════════════════════════════════════════════════════════════╝
↓ everything runs on
╔══════════════════════════════════════════════════════════════════╗
║ PLANE 4 — INFRA (Orca + VMs + Gitea + Infisical + LiteLLM) ║
╚══════════════════════════════════════════════════════════════════╝
╔══════════════════════════════════════════════════════════════════════════════╗
║ PLANE 4 — INFRA cluster: breakpilot-edge VM: vm-edge (co-tenant) ║
║ (Orca + Gitea + Infisical + PowerDNS + Orca-Proxy) ║
╚══════════════════════════════════════════════════════════════════════════════╝
```
Stage and prod share the same App cluster on different VMs (`vm-app-stage` and `vm-app-prod`). Stage authenticates via the prod Identity plane (`auth.breakpilot.com`) and reads tenant config from the prod Control plane (`tenant-registry`) under `tenant.kind = "stage"` — Identity and Control are **not** duplicated for stage.
---
## 4. Plane 1 — Identity
@@ -305,7 +315,9 @@ all real-tenant flows work otherwise same flows, same code paths
---
## 6. Plane 3 — Data
## 6. Plane 3 — App (was "Data")
Renamed 2026-06-30. The plane carries product workloads end-to-end (dashboards, APIs, AI services) as well as the data stores those products own — "App" reflects the full surface, not just the storage tier. SLA targets are unchanged (CERTifAI ≥ 99.5%, compliance ≥ 99.5%, RPO ≤ 6h–24h per data class). Lives in the `breakpilot-app` cluster on `vm-app-prod` (prod) and `vm-app-stage` (stage).
### CERTifAI
@@ -334,10 +346,12 @@ GDPR and AI-Act compliance automation platform. After updates, tenant identity c
## 7. Plane 4 — Infra
**Orchestration:** Orca manages all containers on Hetzner VMs
Co-tenant with the **Identity plane** in the `breakpilot-edge` cluster on `vm-edge`. Single-VM core per the 2026-06-30 cluster-split decision. Keycloak JVM heap is pinned (`-Xmx1500m`) so it cannot starve PowerDNS / Infisical. Escape hatch if the heap fight shows up post-launch: peel Keycloak onto its own VM inside the same cluster — no schema migration, just a `placement.node` flip.
**Orchestration:** Orca manages all containers on SysEleven VMs. Per-node ingress (every node binds 80/443 and runs its own ACME), one Orca cluster per plane (3 clusters total), one IaC repo per cluster (`platform/breakpilot-{edge,control,app}`).
**Secrets:** Infisical — every service has a machine identity, pulls its own secrets at startup
**CI/CD:** Gitea Actions → Docker build → push to private registry → Orca redeploy webhook
**Routing:** Orca-Proxy handles all TLS termination and subdomain routing
**CI/CD:** Gitea Actions → Docker build → push to private registry → Orca redeploy webhook (one CI/release cadence per cluster repo, plus per-product repos for the deployed services)
**Routing:** Orca-Proxy handles TLS termination and subdomain routing on each node (no central proxy SPOF)
```
Orca-Proxy routing table: