platform/docs
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(arch): align INFRASTRUCTURE + PLATFORM_ARCHITECTURE to cluster split #7

Merged
sharang merged 1 commits from refactor/cluster-split-arch-docs into main 2026-06-30 21:30:49 +00:00
Conversation 0 Commits 1 Files Changed 2
+119 -84
2 changed files with 119 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
INFRASTRUCTURE.md
+85 -64
View File
@@ -1,55 +1,59 @@
# Infrastructure Specification # Infrastructure Specification
**Status:** Locked Topology **Status:** Locked Topology
**Authors:** Sharang, Benjamin **Authors:** Sharang, Benjamin
**Date:** 2026-05-11 (topology lock: 2026-05-18) **Date:** 2026-05-11 (topology lock: 2026-05-18; cluster split: 2026-06-30)
**Companion docs:** PLATFORM_ARCHITECTURE.md, IMPLEMENTATION_PLAN.md, COST_PLAN.md **Companion docs:** PLATFORM_ARCHITECTURE.md, IMPLEMENTATION_PLAN.md, COST_PLAN.md
**Cloud provider:** SysEleven Cloud Services (DUS2, OpenStack) **Cloud provider:** SysEleven Cloud Services (DUS2, OpenStack) — multi-VM rollout gated on legal entity
--- ---
> **2026-06-30 cluster-split decision.** The 4 VMs from the May 18 lock are unchanged in count and flavor. What changes is how they're organized: **one Orca cluster per plane**, three cluster repos (`platform/breakpilot-edge`, `platform/breakpilot-control`, `platform/breakpilot-app`). Identity and Infra are co-tenant on `vm-edge` inside `breakpilot-edge` (single-VM core; KC JVM heap pinned so it cannot starve PowerDNS / Infisical). The "Data plane" is renamed to **App plane** to reflect that it carries product workloads, not just data stores. `vm-data` is renamed `vm-app-prod`, and the standalone `stage` VM is renamed `vm-app-stage` — both live in the same `breakpilot-app` cluster. Multi-VM rollout is gated on the legal entity being established so we can sign SysEleven business contracts; until then, single-VM ops continues via `~/workspace/orca-infra`. See `IMPLEMENTATION_PLAN.md §1.1 / M1.2` and `platform/orca-platform/clusters/README.md` for the migration mechanics. **The SLA targets in §6 / §7 below are unaffected** — they apply at the plane level regardless of which cluster a plane lives in.
## 1. VM Inventory ## 1. VM Inventory
**Four billable VMs total.** Three in production (one per plane after collapsing Identity+Infra), one in stage. Dev runs entirely on developer laptops via docker-compose. **Four billable VMs across three Orca clusters.** Dev runs entirely on developer laptops via docker-compose.
``` ```
┌──────────────┬─────────────────┬────────────────────────┬───────────┬─────────────────┐ ┌──────────────────┬────────────────────┬──────────┬────────────────────────┬───────────┬─────────────────┐
│ Name │ Env │ SysEleven flavor │ Public IP │ Planes owned │ │ Name │ Cluster │ Env │ SysEleven flavor │ Public IP │ Planes owned │
├──────────────┼─────────────────┼────────────────────────┼───────────┼─────────────────┤ ├──────────────────┼────────────────────┼──────────┼────────────────────────┼───────────┼─────────────────┤
│ vm-edge │ prod │ m2.small (2v / 8 GB) │ YES (1) │ Identity + Infra│ │ vm-edge │ breakpilot-edge │ prod │ m2.small (2v / 8 GB) │ YES (1) │ Identity + Infra│
│ vm-control │ prod │ m2.medium (4v / 16 GB) │ No │ Control │ │ vm-control │ breakpilot-control │ prod │ m2.medium (4v / 16 GB) │ No │ Control │
│ vm-data │ prod │ m2.medium (4v / 16 GB) │ No │ Data │ │ vm-app-prod │ breakpilot-app │ prod │ m2.medium (4v / 16 GB) │ No │ App (prod) │
│ stage │ stage │ m2.small (2v / 8 GB) │ YES (1) │ App plane only │ │ vm-app-stage │ breakpilot-app │ stage │ m2.small (2v / 8 GB) │ YES (1) │ App (stage) │
│ (dev) │ dev │ local docker-compose │ n/a │ all (in-memory) │ │ (dev) │ — │ dev │ local docker-compose │ n/a │ all (in-memory) │
└──────────────┴─────────────────┴────────────────────────┴───────────┴─────────────────┘ └──────────────────┴────────────────────┴──────────┴────────────────────────┴───────────┴─────────────────┘
``` ```
**Total compute:** 48 GiB-RAM, 12 vCPU. **Monthly compute net: €192 (36M upfront) / €295 (12M) / €435 (On-Demand).** See COST_PLAN.md for the full three-mode table. **Total compute:** 48 GiB-RAM, 12 vCPU. **Monthly compute net: €192 (36M upfront) / €295 (12M) / €435 (On-Demand).** See COST_PLAN.md for the full three-mode table. The cluster split does not change the per-VM bill (same flavors); the additional cost is operational (3 cluster repos to keep CI green in) and is offset by the independent failure-domain wins.
### Why this topology and not the previous 7-VM layout ### Why this topology and not the previous 7-VM layout
The earlier draft proposed one VM per service group (vm-gateway, vm-identity, vm-secrets, vm-ops, vm-control, vm-certifai, vm-compliance). That gave maximum failure isolation but cost 132 GiB-RAM stage+prod. At 5 customers the isolation is unused — every VM ran at <10% utilisation. The locked topology buys back failure isolation incrementally as load grows (see §13 Growth Trajectory). The earlier draft proposed one VM per service group (vm-gateway, vm-identity, vm-secrets, vm-ops, vm-control, vm-certifai, vm-compliance). That gave maximum failure isolation but cost 132 GiB-RAM stage+prod. At 5 customers the isolation is unused — every VM ran at <10% utilisation. The locked topology buys back failure isolation incrementally as load grows (see §13 Growth Trajectory).
Critical isolations preserved even at 4 VMs: Critical isolations preserved even at 4 VMs:
- **vm-edge isolates identity from app workloads.** Keycloak JVM has its own page cache; ERPNext background jobs cannot starve token issuance. - **vm-edge isolates identity from app workloads.** Keycloak JVM has its own page cache + heap pin (`-Xmx1500m`) so ERPNext or product background jobs on a different VM cannot starve token issuance — and PowerDNS / Infisical on the same VM cannot be starved by KC either.
- **vm-data isolates databases from stateless services.** All data-plane DBs share one host, but they're walled off from the portal + ERPNext + Stalwart competing on vm-control. - **vm-app-prod isolates product databases from stateless services.** All app-plane DBs share one host, but they're walled off from the portal + ERPNext + Stalwart competing on vm-control.
- **stage runs the app plane only.** It calls prod Keycloak + prod Tenant Registry under `tenant.kind = stage` rather than mirroring those services. - **vm-app-stage runs the app plane only.** It calls prod Keycloak (`breakpilot-edge`) + prod Tenant Registry (`breakpilot-control`) under `tenant.kind = stage` rather than mirroring those services. Stage and prod live in the **same** `breakpilot-app` Orca cluster on **different** VMs — same config, different physical workloads.
--- ---
## 2. Service-to-VM Mapping ## 2. Service-to-VM Mapping
Each VM is owned by one Orca cluster (see §1). Service manifests live in that cluster's repo at `platform/breakpilot-<name>/services/...`.
``` ```
vm-edge (prod, m2.small 8 GB, public IP) vm-edge (cluster: breakpilot-edge, prod, m2.small 8 GB, public IP)
├── orca-proxy (Orca-managed; wildcard TLS terminator) ├── orca-proxy (Orca-managed; wildcard TLS terminator)
├── powerdns-auth (Orca-managed; authoritative DNS for breakpilot.com) ├── powerdns-auth (Orca-managed; authoritative DNS for breakpilot.com)
├── keycloak-26 (Orca-managed; JVM, ~1.5 GB heap) ├── keycloak-26 (Orca-managed; JVM, ~1.5 GB heap; repurposed from CERTifAI KC at M2.1)
├── postgres-keycloak (Orca-managed; dedicated PG instance for Keycloak only) ├── postgres-keycloak (Orca-managed; dedicated PG instance for Keycloak only)
├── infisical (Orca-managed) ├── infisical (Orca-managed)
├── postgres-infisical (Orca-managed; dedicated PG instance for Infisical only) ├── postgres-infisical (Orca-managed; dedicated PG instance for Infisical only)
├── redis-infisical (Orca-managed; ephemeral) ├── redis-infisical (Orca-managed; ephemeral)
└── gitea (Orca-managed; SQLite backend to avoid a third PG) └── gitea (Orca-managed; SQLite backend to avoid a third PG)
vm-control (prod, m2.medium 16 GB) vm-control (cluster: breakpilot-control, prod, m2.medium 16 GB)
├── customer-portal (Orca-managed; Next.js) ├── customer-portal (Orca-managed; Next.js)
├── tenant-registry (Orca-managed; Go) ├── tenant-registry (Orca-managed; Go)
├── orca-controller (Orca core process; NOT a managed container) ├── orca-controller (Orca core process; NOT a managed container)
@@ -59,7 +63,7 @@ vm-control (prod, m2.medium 16 GB)
├── redis-erpnext (Orca-managed) ├── redis-erpnext (Orca-managed)
└── stalwart-mail (Orca-managed; SMTP/IMAP/JMAP on mail.breakpilot.com) └── stalwart-mail (Orca-managed; SMTP/IMAP/JMAP on mail.breakpilot.com)
vm-data (prod, m2.medium 16 GB) vm-app-prod (cluster: breakpilot-app, prod, m2.medium 16 GB)
├── certifai-dashboard (Orca-managed) ├── certifai-dashboard (Orca-managed)
├── mongodb (Orca-managed) ├── mongodb (Orca-managed)
├── litellm (Orca-managed) ├── litellm (Orca-managed)
@@ -70,22 +74,23 @@ vm-data (prod, m2.medium 16 GB)
├── qdrant (Orca-managed) ├── qdrant (Orca-managed)
└── minio (Orca-managed) └── minio (Orca-managed)
stage (stage, m2.small 8 GB, public IP) vm-app-stage (cluster: breakpilot-app — SAME cluster as vm-app-prod, stage, m2.small 8 GB, public IP)
├── orca-proxy (light; only routes to stage app) ├── orca-proxy (light; only routes to stage app)
├── customer-portal (NEW VERSION under test) ├── customer-portal (NEW VERSION under test)
├── tenant-registry (NEW VERSION under test, talks to ephemeral PG below) ├── tenant-registry (placeholder; stage actually calls PROD tenant-registry — manifest exists for parity, see §5 isolation rules)
├── certifai-dashboard (NEW VERSION under test) ├── certifai-dashboard (NEW VERSION under test)
├── backend-compliance (NEW VERSION under test) ├── backend-compliance (NEW VERSION under test)
├── ai-compliance-sdk (NEW VERSION under test) ├── ai-compliance-sdk (NEW VERSION under test)
├── admin-compliance (NEW VERSION under test) ├── admin-compliance (NEW VERSION under test)
├── litellm (light; same image as prod) ├── litellm (light; same image as prod)
├── postgres-app-stage (ephemeral; lives entirely on stage VM) ├── postgres-app-stage (ephemeral; lives entirely on vm-app-stage)
├── mongodb-stage (ephemeral) ├── mongodb-stage (ephemeral)
└── qdrant-stage (ephemeral, tiny corpus) └── qdrant-stage (ephemeral, tiny corpus)
Calls OUT to prod: Calls OUT to prod:
→ auth.breakpilot.com (Keycloak token issuance, under stage client_id) → auth.breakpilot.com (PROD Keycloak in breakpilot-edge, under stage client_id, tenant.kind = "stage")
→ mail.breakpilot.com (Stalwart SMTP, recipient filter forces +stage@ only) → registry.breakpilot.com (PROD tenant-registry in breakpilot-control, read-only for stage tenants)
→ mail.breakpilot.com (PROD Stalwart in breakpilot-control, recipient filter forces +stage@ only)
→ Polar SANDBOX webhook URL (NEVER prod Polar) → Polar SANDBOX webhook URL (NEVER prod Polar)
→ no calls to prod Postgres-app, MariaDB, MongoDB → no calls to prod Postgres-app, MariaDB, MongoDB
``` ```
@@ -129,7 +134,7 @@ stage (stage, m2.small 8 GB, public IP)
┌────────┴─────────┐ ┌────────┴─────────┐
│ │ │ │
┌──────▼───────┐ ┌───────▼──────┐ ┌──────▼───────┐ ┌───────▼──────┐
│ vm-control │ │ vm-data │ │ vm-control │ │ vm-app-prod │
│ │ │ │ │ │ │ │
│ portal │ │ certifai │ │ portal │ │ certifai │
│ tenant-reg │ │ mongodb │ │ tenant-reg │ │ mongodb │
@@ -166,7 +171,7 @@ Block volumes (Ceph 3x replicated, €0.10/GiB/mo) mounted to each VM.
├──────────────┼───────────────────────────────────────────┼─────────┼─────────────────────┤ ├──────────────┼───────────────────────────────────────────┼─────────┼─────────────────────┤
│ vm-edge │ pg-keycloak + pg-infisical + Gitea repos │ +50 GB │ Slow │ │ vm-edge │ pg-keycloak + pg-infisical + Gitea repos │ +50 GB │ Slow │
│ vm-control │ MariaDB (ERPNext) + Stalwart mail spool │ +250 GB │ Medium │ │ vm-control │ MariaDB (ERPNext) + Stalwart mail spool │ +250 GB │ Medium │
│ vm-data │ MongoDB + pg-app + Qdrant + MinIO │ +500 GB │ Fast (scales w/ N) │ │ vm-app-prod │ MongoDB + pg-app + Qdrant + MinIO │ +500 GB │ Fast (scales w/ N) │
│ stage │ pg-stage + mongo-stage + qdrant-stage │ +50 GB │ Resets per release │ │ stage │ pg-stage + mongo-stage + qdrant-stage │ +50 GB │ Resets per release │
└──────────────┴───────────────────────────────────────────┴─────────┴─────────────────────┘ └──────────────┴───────────────────────────────────────────┴─────────┴─────────────────────┘
@@ -200,10 +205,10 @@ All backups ship to **SysEleven Object Storage** (S3-compatible, geo-redundant D
│ Infisical store │ encrypted → S3 │ Daily │ 30 days │ Infra Plane │ │ Infisical store │ encrypted → S3 │ Daily │ 30 days │ Infra Plane │
│ MariaDB (vm-control) │ mysqldump → S3 │ Every 6h │ 30 days │ Control Plane │ │ MariaDB (vm-control) │ mysqldump → S3 │ Every 6h │ 30 days │ Control Plane │
│ Stalwart queue/store │ tar → S3 │ Daily │ 7 days │ Control Plane │ │ Stalwart queue/store │ tar → S3 │ Daily │ 7 days │ Control Plane │
│ pg-app (vm-data) │ pg_dump → S3-geo │ Every 6h │ 30 days │ Data Plane (owns RPO)│ │ pg-app (vm-app-prod) │ pg_dump → S3-geo │ Every 6h │ 30 days │ App Plane (owns RPO) │
│ MongoDB (vm-data) │ mongodump → S3 │ Daily │ 30 days │ Data Plane │ │ MongoDB (vm-app-prod) │ mongodump → S3 │ Daily │ 30 days │ App Plane │
│ MinIO (vm-data) │ mc mirror → S3 │ Daily │ 90 days │ Data Plane │ │ MinIO (vm-app-prod) │ mc mirror → S3 │ Daily │ 90 days │ App Plane │
│ Qdrant (vm-data) │ API snap → S3 │ Daily │ 14 days │ Data Plane (rebuild) │ │ Qdrant (vm-app-prod) │ API snap → S3 │ Daily │ 14 days │ App Plane (rebuild) │
│ stage * │ no backup │ — │ — │ — (ephemeral) │ │ stage * │ no backup │ — │ — │ — (ephemeral) │
│ Orca config (IaC) │ Gitea (VCS) │ On commit │ Forever │ Infra Plane │ │ Orca config (IaC) │ Gitea (VCS) │ On commit │ Forever │ Infra Plane │
└───────────────────────┴──────────────────┴────────────┴────────────┴──────────────────────┘ └───────────────────────┴──────────────────┴────────────┴────────────┴──────────────────────┘
@@ -254,11 +259,11 @@ STAGE_ISOLATION— stage tenant cannot mutate any prod data; reads-only against
### Plane ownership of constraints ### Plane ownership of constraints
Even though planes now share VMs, the **ownership model is unchanged** — the plane that owns a constraint owns it regardless of which VM hosts the service. The Infra Plane (now collapsed onto vm-edge alongside the Identity plane) still mechanically enforces backup, IaC, secrets, and network constraints. Planes are the **SLA + ownership abstraction**; clusters are the **operational unit**. The plane that owns a constraint owns it regardless of which cluster hosts the service. Identity and Infra are co-tenant in the `breakpilot-edge` cluster on `vm-edge`; they remain distinct planes for purposes of SLA + ownership accounting.
``` ```
╔══════════════════════════════════════════════════════════════════════════════════════════╗ ╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ IDENTITY PLANE (on vm-edge) ║ ║ IDENTITY PLANE (cluster: breakpilot-edge, on vm-edge) ║
║ ║ ║ ║
║ Owns / defines: ║ ║ Owns / defines: ║
║ AVAILABILITY — must be ≥ 99.5% (root dep for everything) ║ ║ AVAILABILITY — must be ≥ 99.5% (root dep for everything) ║
@@ -268,11 +273,16 @@ Even though planes now share VMs, the **ownership model is unchanged** — the p
║ STAGE_ISOLATION— rate-limits stage_client_id; rejects stage JWTs in prod audiences ║ ║ STAGE_ISOLATION— rate-limits stage_client_id; rejects stage JWTs in prod audiences ║
║ ║ ║ ║
║ Co-tenant note: shares vm-edge with Infra Plane services. JVM heap pinned to 1.5 GB ║ ║ Co-tenant note: shares vm-edge with Infra Plane services. JVM heap pinned to 1.5 GB ║
║ in Orca manifest so it cannot starve PowerDNS / Infisical. ║ ║ in Orca manifest (-Xmx1500m) so it cannot starve PowerDNS / Infisical. Escape hatch ║
║ if the heap fight shows up: flip placement.node to a new vm-edge-identity VM inside ║
║ the SAME breakpilot-edge cluster — no schema migration, just a container move. ║
║ ║
║ KC instance is REPURPOSED from the existing CERTifAI Keycloak at M2.1 (realm export ║
║ + user database move; hostname swap), not stood up fresh. ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝ ╚══════════════════════════════════════════════════════════════════════════════════════════╝
╔══════════════════════════════════════════════════════════════════════════════════════════╗ ╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ CONTROL PLANE (on vm-control) ║ ║ CONTROL PLANE (cluster: breakpilot-control, on vm-control) ║
║ ║ ║ ║
║ Owns / defines: ║ ║ Owns / defines: ║
║ RPO (tenant) — tenant registry & compliance schemas RPO ≤ 6h ║ ║ RPO (tenant) — tenant registry & compliance schemas RPO ≤ 6h ║
@@ -288,10 +298,13 @@ Even though planes now share VMs, the **ownership model is unchanged** — the p
║ mariadb: 3 GB memory cap ║ ║ mariadb: 3 GB memory cap ║
║ stalwart: 1 GB memory cap ║ ║ stalwart: 1 GB memory cap ║
║ tenant-registry: 500 MB ║ ║ tenant-registry: 500 MB ║
║ ║
║ Stage callers: vm-app-stage in breakpilot-app calls THIS tenant-registry under ║
║ tenant.kind = "stage" (read-only for stage tenants). No duplicated tenant-registry. ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝ ╚══════════════════════════════════════════════════════════════════════════════════════════╝
╔══════════════════════════════════════════════════════════════════════════════════════════╗ ╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ DATA PLANE (on vm-data) ║ ║ APP PLANE (was "Data Plane"; cluster: breakpilot-app, on vm-app-prod + vm-app-stage) ║
║ ║ ║ ║
║ Owns / defines: ║ ║ Owns / defines: ║
║ DATA_RESIDENCY — all customer data (MongoDB, pg-app, MinIO) must stay EU ║ ║ DATA_RESIDENCY — all customer data (MongoDB, pg-app, MinIO) must stay EU ║
@@ -300,12 +313,17 @@ Even though planes now share VMs, the **ownership model is unchanged** — the p
║ AUDIT_TRAIL — product-level actions ║ ║ AUDIT_TRAIL — product-level actions ║
║ AVAILABILITY — CERTifAI ≥ 99.5%; compliance ≥ 99.5% ║ ║ AVAILABILITY — CERTifAI ≥ 99.5%; compliance ≥ 99.5% ║
║ ║ ║ ║
║ Co-tenant note: this VM is the SCALE driver. When vm-data hits 80% RAM, bump flavor ║ ║ Co-tenant note: vm-app-prod is the SCALE driver. When it hits 80% RAM, bump flavor ║
║ (m2.medium → m2.large → m2.xlarge). See §13 Growth Trajectory. ║ ║ (m2.medium → m2.large → m2.xlarge). See §13 Growth Trajectory. ║
║ ║
║ Stage VM (vm-app-stage) lives in the SAME cluster — same config, different physical ║
║ workload. Stage authenticates against the prod Identity plane (breakpilot-edge KC) ║
║ and reads tenant config from the prod Control plane (breakpilot-control ║
║ tenant-registry) — never duplicates identity or control. ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝ ╚══════════════════════════════════════════════════════════════════════════════════════════╝
╔══════════════════════════════════════════════════════════════════════════════════════════╗ ╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ INFRA PLANE (on vm-edge, alongside Identity) ║ ║ INFRA PLANE (cluster: breakpilot-edge, on vm-edge, alongside Identity) ║
║ ║ ║ ║
║ Owns / enforces ALL of: ║ ║ Owns / enforces ALL of: ║
║ BACKUP — executes all backup jobs (pg_dump, mongodump, mc mirror) ║ ║ BACKUP — executes all backup jobs (pg_dump, mongodump, mc mirror) ║
@@ -317,6 +335,9 @@ Even though planes now share VMs, the **ownership model is unchanged** — the p
║ AVAILABILITY — Orca restart policies, health checks ║ ║ AVAILABILITY — Orca restart policies, health checks ║
║ COLD_START — enforces startup ordering (see §10 Scenario F) ║ ║ COLD_START — enforces startup ordering (see §10 Scenario F) ║
║ STAGE_ISOLATION— Infisical secret-path scoping for stage_app identity ║ ║ STAGE_ISOLATION— Infisical secret-path scoping for stage_app identity ║
║ ║
║ Per-cluster note: each of the three cluster repos has its own [backup] block + S3 ║
║ bucket. A runaway backup on breakpilot-app cannot fill breakpilot-edge's bucket. ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝ ╚══════════════════════════════════════════════════════════════════════════════════════════╝
``` ```
@@ -339,14 +360,14 @@ Even though planes now share VMs, the **ownership model is unchanged** — the p
│ Frappe HD │ 99% │ 60 min │ 24h │ vm-control │ │ Frappe HD │ 99% │ 60 min │ 24h │ vm-control │
│ MariaDB │ 99.5% │ 20 min │ 6h │ vm-control │ │ MariaDB │ 99.5% │ 20 min │ 6h │ vm-control │
│ Stalwart Mail │ 99% │ 60 min │ 24h │ vm-control │ │ Stalwart Mail │ 99% │ 60 min │ 24h │ vm-control │
│ CERTifAI │ 99.5% │ 10 min │ 24h │ vm-data │ │ CERTifAI │ 99.5% │ 10 min │ 24h │ vm-app-prod │
│ MongoDB │ 99.5% │ 20 min │ 24h │ vm-data │ │ MongoDB │ 99.5% │ 20 min │ 24h │ vm-app-prod │
│ LiteLLM │ 99% │ 5 min │ N/A │ vm-data │ │ LiteLLM │ 99% │ 5 min │ N/A │ vm-app-prod │
│ backend-compliance │ 99.5% │ 10 min │ 6h │ vm-data │ │ backend-compliance │ 99.5% │ 10 min │ 6h │ vm-app-prod │
│ ai-compliance-sdk │ 99.5% │ 10 min │ 6h │ vm-data │ │ ai-compliance-sdk │ 99.5% │ 10 min │ 6h │ vm-app-prod │
│ pg-app │ 99.9% │ 20 min │ 6h │ vm-data (SPOF — RISK-1) │ │ pg-app │ 99.9% │ 20 min │ 6h │ vm-app-prod (SPOF — RISK-1) │
│ MinIO │ 99.5% │ 30 min │ 24h │ vm-data │ │ MinIO │ 99.5% │ 30 min │ 24h │ vm-app-prod │
│ Qdrant │ 99% │ 2h │ 24h │ vm-data (rebuildable) │ │ Qdrant │ 99% │ 2h │ 24h │ vm-app-prod (rebuildable) │
│ stage (any service) │ 95% │ best ef.│ N/A │ stage (ephemeral; no SLA) │ │ stage (any service) │ 95% │ best ef.│ N/A │ stage (ephemeral; no SLA) │
└───────────────────────┴──────────────┴─────────┴─────────┴────────────────────────────────┘ └───────────────────────┴──────────────┴─────────┴─────────┴────────────────────────────────┘
``` ```
@@ -433,13 +454,13 @@ Arrows = "requires to function." Dashed = soft (degrades, doesn't fail).
│ │ mariadb + redis-erp ──► erpnext + frappe-hd │ │ │ │ mariadb + redis-erp ──► erpnext + frappe-hd │ │
│ │ (intra) ─────────────► stalwart │ │ │ │ (intra) ─────────────► stalwart │ │
│ │ ──────────────────────► customer-portal │ │ │ │ ──────────────────────► customer-portal │ │
│ │ ──────────────────────► tenant-registry ──► pg-app (vm-data)│ │ │ │ ──────────────────────► tenant-registry ──► pg-app (vm-app-prod)│ │
│ └────────────────────────────────────────────────────────────┘ │ │ └────────────────────────────────────────────────────────────┘ │
│ │ tenant-registry API │ │ │ tenant-registry API │
└────────────────────────────┼─────────────────────────────────────┘ └────────────────────────────┼─────────────────────────────────────┘
▼ ▼
┌──────────────────────────────────────────────────────────────────┐ ┌──────────────────────────────────────────────────────────────────┐
│ vm-data (Data) │ │ vm-app-prod (Data) │
│ ┌────────────────────────────────────────────────────────────┐ │ │ ┌────────────────────────────────────────────────────────────┐ │
│ │ mongodb ───► certifai ◄── (vm-edge JWKS, vm-edge secrets) │ │ │ │ mongodb ───► certifai ◄── (vm-edge JWKS, vm-edge secrets) │ │
│ │ litellm ───► certifai, ai-compliance-sdk │ │ │ │ litellm ───► certifai, ai-compliance-sdk │ │
@@ -470,9 +491,9 @@ Arrows = "requires to function." Dashed = soft (degrades, doesn't fail).
├──► keycloak (vm-edge) ──► pg-keycloak (intra-VM) ├──► keycloak (vm-edge) ──► pg-keycloak (intra-VM)
│ │
└──► customer-portal (vm-control) └──► customer-portal (vm-control)
├──► tenant-registry (vm-control) ──► pg-app (vm-data) ├──► tenant-registry (vm-control) ──► pg-app (vm-app-prod)
├──► certifai (vm-data) ──► mongodb (intra-VM) ├──► certifai (vm-app-prod) ──► mongodb (intra-VM)
└──► backend-compliance (vm-data) ──► pg-app (intra-VM) └──► backend-compliance (vm-app-prod) ──► pg-app (intra-VM)
──► ai-sdk ──► qdrant + minio ──► ai-sdk ──► qdrant + minio
──► litellm ──► [external AI APIs] ──► litellm ──► [external AI APIs]
``` ```
@@ -515,7 +536,7 @@ Impact: customer-portal: DOWN → /[slug]/* all return 503
ERPNext + Frappe HD: DOWN → we cannot create sales orders or read tickets ERPNext + Frappe HD: DOWN → we cannot create sales orders or read tickets
Stalwart: DOWN → no outbound emails (trial nudges, exports, ticket replies) Stalwart: DOWN → no outbound emails (trial nudges, exports, ticket replies)
MariaDB: DOWN → ERPNext queries fail; backups paused MariaDB: DOWN → ERPNext queries fail; backups paused
Products (CERTifAI, compliance): UNAFFECTED (on vm-data, JWTs still validate) Products (CERTifAI, compliance): UNAFFECTED (on vm-app-prod, JWTs still validate)
Existing logged-in users: can use products directly via product subdomain Existing logged-in users: can use products directly via product subdomain
IF they bookmark it; portal home is 503. IF they bookmark it; portal home is 503.
Cascade: T+0: portal 503; new tenant onboarding blocked (registry down) Cascade: T+0: portal 503; new tenant onboarding blocked (registry down)
@@ -532,7 +553,7 @@ Cost of fix at Tier B/C: split vm-control → vm-portal + vm-ops (ERPNext)
— €64/mo extra at m2.small — €64/mo extra at m2.small
``` ```
### Scenario C — vm-data fails ### Scenario C — vm-app-prod fails
``` ```
Impact: tenant-registry queries: FAIL (pg-app down) → portal returns 503 for tenant lookup Impact: tenant-registry queries: FAIL (pg-app down) → portal returns 503 for tenant lookup
@@ -543,7 +564,7 @@ Impact: tenant-registry queries: FAIL (pg-app down) → portal returns 503 fo
Cascade: T+0: products down; portal degraded Cascade: T+0: products down; portal degraded
T+15m: support tickets pile up T+15m: support tickets pile up
Note: prod is partial — users see error pages but ERPNext + auth still work Note: prod is partial — users see error pages but ERPNext + auth still work
Recovery: Restart vm-data containers. If pg-app corrupt: restore from pg_dump (RPO 6h). Recovery: Restart vm-app-prod containers. If pg-app corrupt: restore from pg_dump (RPO 6h).
RTO target: 20 min RTO target: 20 min
Mitigation: This is the SCALE-event VM. RISK-1 below makes this the worst SPOF: Mitigation: This is the SCALE-event VM. RISK-1 below makes this the worst SPOF:
one pg-app instance owns tenant_registry + compliance schemas. one pg-app instance owns tenant_registry + compliance schemas.
@@ -561,11 +582,11 @@ Impact: CERTifAI: AI features fail (summarization, chat completion).
Compliance CRUD: UNAFFECTED. Compliance CRUD: UNAFFECTED.
Cascade: Soft degradation only. Products show "AI features temporarily unavailable" banner. Cascade: Soft degradation only. Products show "AI features temporarily unavailable" banner.
Deadlock: None. Deadlock: None.
Recovery: Restart LiteLLM on vm-data (stateless, ~30s). Recovery: Restart LiteLLM on vm-app-prod (stateless, ~30s).
Severity: MEDIUM — graceful degradation by design Severity: MEDIUM — graceful degradation by design
``` ```
### Scenario E — Stage VM compromised or buggy ### Scenario E — vm-app-stage compromised or buggy
``` ```
Impact: On stage itself: stage portal serves bad data; stage testers see errors. Impact: On stage itself: stage portal serves bad data; stage testers see errors.
@@ -587,7 +608,7 @@ Severity: LOW (in prod) / HIGH (on stage, but stage SLA is 95%)
Three VMs boot at once. Services must start in dependency order or services Three VMs boot at once. Services must start in dependency order or services
crash-loop until their deps are ready. crash-loop until their deps are ready.
DEADLOCK RISK: vm-control services (portal, tenant-registry) start before vm-data DEADLOCK RISK: vm-control services (portal, tenant-registry) start before vm-app-prod
services (pg-app, certifai, compliance). They'll crash-loop ~2-5min services (pg-app, certifai, compliance). They'll crash-loop ~2-5min
with backoff retries. with backoff retries.
Same for ERPNext on vm-control trying to reach Keycloak on vm-edge. Same for ERPNext on vm-control trying to reach Keycloak on vm-edge.
@@ -598,7 +619,7 @@ RESOLUTION: Orca enforces cross-VM startup ordering via health-check dependencie
Required cold start sequence: Required cold start sequence:
Phase 0 — Data roots on vm-data (parallel): Phase 0 — Data roots on vm-app-prod (parallel):
pg-app, mongodb, qdrant, minio pg-app, mongodb, qdrant, minio
Phase 0 — Data roots on vm-control (parallel): Phase 0 — Data roots on vm-control (parallel):
mariadb, redis-erpnext mariadb, redis-erpnext
@@ -613,7 +634,7 @@ Required cold start sequence:
keycloak (needs: pg-keycloak [Phase 0], infisical [Phase 1]) keycloak (needs: pg-keycloak [Phase 0], infisical [Phase 1])
gitea (needs: sqlite; ready from Phase 0) gitea (needs: sqlite; ready from Phase 0)
Phase 3 — Control on vm-control + Data services on vm-data (parallel): Phase 3 — Control on vm-control + Data services on vm-app-prod (parallel):
tenant-registry (needs: keycloak [Phase 2], pg-app [Phase 0, remote]) tenant-registry (needs: keycloak [Phase 2], pg-app [Phase 0, remote])
erpnext + frappe-hd (needs: mariadb, redis-erpnext [Phase 0], keycloak [Phase 2]) erpnext + frappe-hd (needs: mariadb, redis-erpnext [Phase 0], keycloak [Phase 2])
stalwart (needs: infisical [Phase 1]) stalwart (needs: infisical [Phase 1])
@@ -642,7 +663,7 @@ Impact: Portal cannot resolve tenant from subdomain → /[slug]/* all 503
Cascade: New logins degraded. Cascade: New logins degraded.
Existing sessions continue. Existing sessions continue.
Deadlock: None. Deadlock: None.
Recovery: Restart tenant-registry on vm-control. pg-app on vm-data must be healthy. Recovery: Restart tenant-registry on vm-control. pg-app on vm-app-prod must be healthy.
RTO target: ≤ 60s RTO target: ≤ 60s
Mitigation: Portal caches slug → tenant mapping with 60s TTL. Mitigation: Portal caches slug → tenant mapping with 60s TTL.
Short outage invisible to customers. Short outage invisible to customers.
@@ -683,7 +704,7 @@ stage-app │ │ │ │ │ │ │
## 12. Open Infrastructure Risks (Priority Order) ## 12. Open Infrastructure Risks (Priority Order)
``` ```
RISK-1 pg-app (vm-data) is a single instance serving tenant_registry + compliance schemas. RISK-1 pg-app (vm-app-prod) is a single instance serving tenant_registry + compliance schemas.
One crash blocks portal AND compliance product simultaneously. One crash blocks portal AND compliance product simultaneously.
→ Mitigation: split into pg-registry + pg-compliance at Tier B (200 customers). → Mitigation: split into pg-registry + pg-compliance at Tier B (200 customers).
Move pg-registry to its own DBaaS PostgreSQL cluster (€213/mo). Move pg-registry to its own DBaaS PostgreSQL cluster (€213/mo).
@@ -738,10 +759,10 @@ The locked 4-VM topology is right for 5–~200 customers. Past that, expect to a
``` ```
Tier A (5–200 cust): 4 VMs as locked €192/mo compute (36M upfront) Tier A (5–200 cust): 4 VMs as locked €192/mo compute (36M upfront)
↓ ↓
Tier B (200–500): Bump vm-data m2.med → m2.large +€64/mo Tier B (200–500): Bump vm-app-prod m2.med → m2.large +€64/mo
Add cold-standby vm-edge-spare +€0 (idle, paid only on swap) Add cold-standby vm-edge-spare +€0 (idle, paid only on swap)
↓ ↓
Tier C (500–1000): Split vm-data: vm-data + vm-data-db +€64/mo Tier C (500–1000): Split vm-app-prod: vm-app-prod + vm-app-prod-db +€64/mo
(postgres-app moves to its own VM, or DBaaS cluster +€213/mo) (postgres-app moves to its own VM, or DBaaS cluster +€213/mo)
Split vm-control: vm-control + vm-ops +€64/mo Split vm-control: vm-control + vm-ops +€64/mo
(ERPNext + MariaDB + Stalwart move to vm-ops) (ERPNext + MariaDB + Stalwart move to vm-ops)
@@ -749,7 +770,7 @@ Tier C (500–1000): Split vm-data: vm-data + vm-data-db +€64/mo
Tier D (1000–2000): Split vm-edge: vm-edge + vm-identity + vm-secrets +€96/mo Tier D (1000–2000): Split vm-edge: vm-edge + vm-identity + vm-secrets +€96/mo
HA Keycloak active-passive on 2× vm-identity +€32/mo HA Keycloak active-passive on 2× vm-identity +€32/mo
Octavia Load Balancer Double Instance +€58/mo Octavia Load Balancer Double Instance +€58/mo
vm-data m2.large → m2.xlarge or 2× +€128–256/mo vm-app-prod m2.large → m2.xlarge or 2× +€128–256/mo
↓ ↓
Final topology ≈ 8 prod VMs + DBaaS Final topology ≈ 8 prod VMs + DBaaS
``` ```
PLATFORM_ARCHITECTURE.md
+34 -20
View File
@@ -1,10 +1,12 @@
# Platform Architecture — B2B Customer Portal # Platform Architecture — B2B Customer Portal
**Status:** Design Draft **Status:** Design Draft
**Authors:** Sharang, Benjamin **Authors:** Sharang, Benjamin
**Date:** 2026-05-11 **Date:** 2026-05-11 (cluster split: 2026-06-30)
--- ---
> **2026-06-30 cluster-split decision.** The four-plane vocabulary in this document still applies as the **SLA + ownership abstraction**. What changed is the operational shape: planes are now hosted by **three Orca clusters** (one cluster per plane, with Identity and Infra co-tenant in the same cluster on a single VM). The "Data plane" is renamed **App plane** to reflect that it carries product workloads, not just data stores. Cluster repos: `platform/breakpilot-edge` (Identity + Infra), `platform/breakpilot-control` (Control), `platform/breakpilot-app` (App, with prod + stage VMs). Multi-VM rollout is gated on the legal entity being established. See `INFRASTRUCTURE.md §1`, `IMPLEMENTATION_PLAN.md §1.1 / M1.2`, and `platform/orca-platform/clusters/README.md` for the operational details.
## 1. Vision ## 1. Vision
We sell CERTifAI and breakpilot-compliance as modular B2B building blocks. Customers buy one or both and operate them inside a unified customer portal — without needing to understand that they are separate products under the hood. We sell CERTifAI and breakpilot-compliance as modular B2B building blocks. Customers buy one or both and operate them inside a unified customer portal — without needing to understand that they are separate products under the hood.
@@ -28,24 +30,32 @@ Out of scope: breakpilot-dataroom, breakpilot-lehrer, breakpilot-pitch-deck.
## 3. The Four Planes ## 3. The Four Planes
Four logical planes — but only **three Orca clusters**. Identity and Infra are co-tenant in the `breakpilot-edge` cluster on `vm-edge` per the 2026-06-30 cluster-split decision. The plane abstraction owns SLA and audit responsibilities; the cluster abstraction owns operational concerns (failure domain, backup policy, CI scope, IaC repo).
``` ```
╔══════════════════════════════════════════════════════════════════╗ ╔══════════════════════════════════════════════════════════════════════════════╗
║ PLANE 1 — IDENTITY (logical root, all auth flows through here) ║ ║ PLANE 1 — IDENTITY cluster: breakpilot-edge VM: vm-edge ║
╚══════════════════════════════════════════════════════════════════╝ ║ (logical root, all auth flows through here) ║
╚══════════════════════════════════════════════════════════════════════════════╝
↓ JWT ↓ JWT
╔══════════════════════════════════════════════════════════════════╗ ╔══════════════════════════════════════════════════════════════════════════════╗
║ PLANE 2 — CONTROL (portal + ERPNext + tenant registry) ║ ║ PLANE 2 — CONTROL cluster: breakpilot-control VM: vm-control ║
╚══════════════════════════════════════════════════════════════════╝ ║ (portal + ERPNext + tenant registry) ║
╚══════════════════════════════════════════════════════════════════════════════╝
↓ tenant-scoped API calls ↓ tenant-scoped API calls
╔══════════════════════════════════════════════════════════════════╗ ╔══════════════════════════════════════════════════════════════════════════════╗
║ PLANE 3 — DATA (CERTifAI + breakpilot-compliance) ║ ║ PLANE 3 — APP (was DATA) cluster: breakpilot-app VMs: vm-app-prod ║
╚══════════════════════════════════════════════════════════════════╝ ║ (CERTifAI + breakpilot-compliance + data stores) + vm-app-stage║
╚══════════════════════════════════════════════════════════════════════════════╝
↓ everything runs on ↓ everything runs on
╔══════════════════════════════════════════════════════════════════╗ ╔══════════════════════════════════════════════════════════════════════════════╗
║ PLANE 4 — INFRA (Orca + VMs + Gitea + Infisical + LiteLLM) ║ ║ PLANE 4 — INFRA cluster: breakpilot-edge VM: vm-edge (co-tenant) ║
╚══════════════════════════════════════════════════════════════════╝ ║ (Orca + Gitea + Infisical + PowerDNS + Orca-Proxy) ║
╚══════════════════════════════════════════════════════════════════════════════╝
``` ```
Stage and prod share the same App cluster on different VMs (`vm-app-stage` and `vm-app-prod`). Stage authenticates via the prod Identity plane (`auth.breakpilot.com`) and reads tenant config from the prod Control plane (`tenant-registry`) under `tenant.kind = "stage"` — Identity and Control are **not** duplicated for stage.
--- ---
## 4. Plane 1 — Identity ## 4. Plane 1 — Identity
@@ -305,7 +315,9 @@ all real-tenant flows work otherwise same flows, same code paths
--- ---
## 6. Plane 3 — Data ## 6. Plane 3 — App (was "Data")
Renamed 2026-06-30. The plane carries product workloads end-to-end (dashboards, APIs, AI services) as well as the data stores those products own — "App" reflects the full surface, not just the storage tier. SLA targets are unchanged (CERTifAI ≥ 99.5%, compliance ≥ 99.5%, RPO ≤ 6h–24h per data class). Lives in the `breakpilot-app` cluster on `vm-app-prod` (prod) and `vm-app-stage` (stage).
### CERTifAI ### CERTifAI
@@ -334,10 +346,12 @@ GDPR and AI-Act compliance automation platform. After updates, tenant identity c
## 7. Plane 4 — Infra ## 7. Plane 4 — Infra
**Orchestration:** Orca manages all containers on Hetzner VMs Co-tenant with the **Identity plane** in the `breakpilot-edge` cluster on `vm-edge`. Single-VM core per the 2026-06-30 cluster-split decision. Keycloak JVM heap is pinned (`-Xmx1500m`) so it cannot starve PowerDNS / Infisical. Escape hatch if the heap fight shows up post-launch: peel Keycloak onto its own VM inside the same cluster — no schema migration, just a `placement.node` flip.
**Secrets:** Infisical — every service has a machine identity, pulls its own secrets at startup
**CI/CD:** Gitea Actions → Docker build → push to private registry → Orca redeploy webhook **Orchestration:** Orca manages all containers on SysEleven VMs. Per-node ingress (every node binds 80/443 and runs its own ACME), one Orca cluster per plane (3 clusters total), one IaC repo per cluster (`platform/breakpilot-{edge,control,app}`).
**Routing:** Orca-Proxy handles all TLS termination and subdomain routing **Secrets:** Infisical — every service has a machine identity, pulls its own secrets at startup
**CI/CD:** Gitea Actions → Docker build → push to private registry → Orca redeploy webhook (one CI/release cadence per cluster repo, plus per-product repos for the deployed services)
**Routing:** Orca-Proxy handles TLS termination and subdomain routing on each node (no central proxy SPOF)
``` ```
Orca-Proxy routing table: Orca-Proxy routing table: