484 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Benjamin Admin	49ce417428	feat: add compliance modules 2-5 (dashboard, security templates, process manager, evidence collector) ... Module 2: Extended Compliance Dashboard with roadmap, module-status, next-actions, snapshots, score-history Module 3: 7 German security document templates (IT-Sicherheitskonzept, Datenschutz, Backup, Logging, Incident-Response, Zugriff, Risikomanagement) Module 4: Compliance Process Manager with CRUD, complete/skip/seed, ~50 seed tasks, 3-tab UI Module 5: Evidence Collector Extended with automated checks, control-mapping, coverage report, 4-tab UI Also includes: canonical control library enhancements (verification method, categories, dedup), control generator improvements, RAG client extensions 52 tests pass, frontend builds clean. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-14 21:03:04 +01:00
Benjamin Admin	13d13c8226	fix: add all RAG regulation codes to license mapping ... Many regulation codes (nist_sp800_53r5, eucsa, owasp_top10_2021, EDPB guidelines, EU laws, AT/FR/ES/NL/IT/HU laws) were defaulting to Rule 3 (restricted) because they weren't in REGULATION_LICENSE_MAP. Now all ~100 regulation codes from RAG are properly mapped to Rule 1 or 2. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-14 08:38:31 +01:00
Benjamin Admin	b6e6ffaaee	feat: add verification method, categories, and dedup UI to control library ... - Migration 047: verification_method + category columns, 17 category lookup table - Backend: new filters, GET /categories, GET /controls/{id}/similar (embedding-based) - Frontend: filter dropdowns, badges, dedup UI in ControlDetail with merge workflow - ControlForm: verification method + category selects - Provenance: verification methods, categories, master library strategy sections - Fix UUID cast syntax in generator routes (::uuid -> CAST) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-14 07:55:22 +01:00
Benjamin Admin	8a05fcc2f0	refactor: split control library into components, add generator UI ... - Extract ControlForm, ControlDetail, GeneratorModal, helpers into separate component files (max ~470 lines each, was 1210) - Add Collection selector in Generator modal - Add Job History view in Generator modal - Add Review Queue button with counter badge - Add review mode navigation (prev/next through review items) - Add vitest tests for helpers (getDomain, constants, options) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 18:52:42 +01:00
Benjamin Admin	9812ff46f3	feat: add 7-stage control generator pipeline with 3 license rules ... - control_generator.py: RAG→License→Structure/Reform→Harmonize→Anchor→Store→Mark pipeline with Anthropic Claude API (primary) + Ollama fallback for LLM reformulation - anchor_finder.py: RAG-based + DuckDuckGo anchor search for open references - control_generator_routes.py: REST API for generate, job status, review queue, processed stats - 046_control_generator.sql: job tracking, chunk tracking, blocked sources tables; extends canonical_controls with license_rule, source_original_text, source_citation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 18:42:40 +01:00
Benjamin Admin	30236c0001	docs: add post-push deploy monitoring to CLAUDE.md ... After every push to gitea, Claude now automatically polls health endpoints and notifies the user when the deployment is ready for testing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 13:39:12 +01:00
Benjamin Admin	b4d2be83eb	Merge gitea/main: resolve ci.yaml conflict, keep Coolify deploy ... Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 13:26:17 +01:00
Benjamin Admin	38c7cf0a00	Merge branch 'main' of ssh://gitea.meghsakha.com:22222/Benjamin_Boenisch/breakpilot-compliance	2026-03-13 13:23:30 +01:00
Benjamin Admin	399fa62267	docs: update all docs to reflect Coolify deployment model ... Replace Hetzner references with Coolify. Deployment is now: - Core + Compliance: Push gitea → Coolify auto-deploys - Lehrer: stays local on Mac Mini Updated: CLAUDE.md, MkDocs CI/CD pipeline, MkDocs index. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 12:09:51 +01:00
Sharang Parnerkar	f1710fdb9e	fix: migrate deployment from Hetzner to Coolify (#1) ... ## Summary - Add Coolify deployment configuration (docker-compose, healthchecks, network setup) - Replace deploy-hetzner CI job with Coolify webhook deploy - Externalize postgres, qdrant, S3 for Coolify environment ## All changes since branch creation - Coolify docker-compose with Traefik labels and healthchecks - CI pipeline: deploy-hetzner → deploy-coolify (simple webhook curl) - SQLAlchemy 2.x text() compatibility fixes - Alpine-compatible Dockerfile fixes Co-authored-by: Sharang Parnerkar <parnerkarsharang@gmail.com> Reviewed-on: #1	2026-03-13 10:45:35 +00:00
Sharang Parnerkar	1dfea51919	Remove standalone deploy-coolify.yml — deploy is handled in ci.yaml ... Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 11:26:31 +01:00
Sharang Parnerkar	559d7960a2	Replace deploy-hetzner with Coolify webhook deploy ... Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 10:39:12 +01:00
Sharang Parnerkar	a101426dba	Add traefik.docker.network label to fix routing ... Containers are on multiple networks (breakpilot-network, coolify, gokocgws...). Without traefik.docker.network, Traefik randomly picks a network and may choose breakpilot-network where it has no access. This label forces Traefik to always use the coolify network. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 10:16:52 +01:00
Sharang Parnerkar	f6b22820ce	Add coolify network to externally-routed services ... Traefik routes traffic via the 'coolify' bridge network, so services that need public domain access must be on both breakpilot-network (for inter-service communication) and coolify (for Traefik routing). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 10:16:52 +01:00
Sharang Parnerkar	86588aff09	Fix SQLAlchemy 2.x compatibility: wrap raw SQL in text() ... SQLAlchemy 2.x requires raw SQL strings to be explicitly wrapped in text(). Fixed 16 instances across 5 route files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 10:16:52 +01:00
Sharang Parnerkar	033fa52e5b	Add healthcheck to dsms-gateway ... Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 10:16:00 +01:00
Sharang Parnerkar	005fb9d219	Add healthchecks to admin-compliance, developer-portal, backend-compliance ... Traefik may require healthchecks to route traffic to containers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 10:16:00 +01:00
Sharang Parnerkar	0c01f1c96c	Remove Traefik labels from coolify compose — Coolify handles routing ... Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 10:16:00 +01:00
Sharang Parnerkar	ffd256d420	Sync coolify compose with main: use COMPLIANCE_DATABASE_URL, QDRANT_URL ... - Switch to ${COMPLIANCE_DATABASE_URL} for admin-compliance, backend, SDK, crawler - Add DATABASE_URL to admin-compliance environment - Switch ai-compliance-sdk from QDRANT_HOST/PORT to QDRANT_URL + QDRANT_API_KEY - Add MINIO_SECURE to compliance-tts-service - Update .env.coolify.example with new variable patterns Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 10:16:00 +01:00
Sharang Parnerkar	d542dbbacd	fix: ensure public dir exists in developer-portal build ... Next.js standalone COPY fails when no public directory exists in source. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 10:16:00 +01:00
Sharang Parnerkar	a3d0024d39	fix: use Alpine-compatible addgroup/adduser flags in Dockerfiles ... Replace --system/--gid/--uid (Debian syntax) with -S/-g/-u (BusyBox/Alpine). Coolify ARG injection causes exit code 255 with Debian-style flags. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 10:13:57 +01:00
Sharang Parnerkar	998d427c3c	fix: update alpine base to 3.21 for ai-compliance-sdk ... Alpine 3.19 apk mirrors failing during Coolify build. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 10:13:57 +01:00
Sharang Parnerkar	99f3180ffc	refactor(coolify): externalize postgres, qdrant, S3 ... - Replace bp-core-postgres with POSTGRES_HOST env var - Replace bp-core-qdrant with QDRANT_HOST env var - Replace bp-core-minio with S3_ENDPOINT/S3_ACCESS_KEY/S3_SECRET_KEY Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 10:13:57 +01:00
Sharang Parnerkar	2ec340c64b	feat: add Coolify deployment configuration ... Add docker-compose.coolify.yml (8 services), .env.coolify.example, and Gitea Action workflow for Coolify API deployment. Removes core-health-check and docs. Adds Traefik labels for *.breakpilot.ai domain routing with Let's Encrypt SSL. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 10:13:57 +01:00
Benjamin Admin	499ddc04d5	chore: trigger redeploy via Gitea Actions CI/CD ... Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 09:54:23 +01:00
Benjamin Admin	f738ca8c52	fix: make compliance router imports resilient to individual module failures ... Replaced bare imports with safe_import_router pattern — if one sub-router fails to import (e.g. missing dependency), other routers still load. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 09:46:52 +01:00
Benjamin Admin	c530898963	fix: replace Python 3.10+ union type syntax with typing.Optional for Pydantic v2 compat ... from __future__ import annotations breaks Pydantic BaseModel runtime type evaluation. Replaced str | None → Optional[str], list[str] → List[str] etc. in control_generator.py, anchor_finder.py, control_generator_routes.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 09:36:14 +01:00
Benjamin Admin	cdafc4d9f4	feat: auto-run SQL migrations on backend startup ... Adds migration_runner.py that executes pending migrations from migrations/ directory when backend-compliance starts. Tracks applied migrations in _migration_history table. Handles existing databases: detects if tables from migrations 001-045 already exist and seeds the history table accordingly, so only new migrations (046+) are applied. Skippable via SKIP_MIGRATIONS=true env var. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 09:14:18 +01:00
Benjamin Admin	de19ef0684	feat(control-generator): 7-stage pipeline for RAG→LLM→Controls generation ... Implements the Control Generator Pipeline that systematically generates canonical security controls from 150k+ RAG chunks across all compliance collections (BSI, NIST, OWASP, ENISA, EU laws, German laws). Three license rules enforced throughout: - Rule 1 (free_use): Laws/Public Domain — original text preserved - Rule 2 (citation_required): CC-BY/CC-BY-SA — text with citation - Rule 3 (restricted): BSI/ISO — full reformulation, no source traces New files: - Migration 046: job tracking, chunk tracking, blocked sources tables - control_generator.py: 7-stage pipeline (scan→classify→structure/reform→harmonize→anchor→store→mark) - anchor_finder.py: RAG + DuckDuckGo open-source reference search - control_generator_routes.py: REST API (generate, review, stats, blocked-sources) - test_control_generator.py: license mapping, rule enforcement, anchor filtering tests Modified: - __init__.py: register control_generator_router - route.ts: proxy generator/review/stats endpoints - page.tsx: Generator modal, stats panel, state filter, review queue, license badges Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 09:03:37 +01:00
Benjamin Admin	c87f07c99a	feat: seed 10 canonical controls + CRUD endpoints + frontend editor ... - Migration 045: Seed 10 controls (AUTH, NET, SUP, LOG, WEB, DATA, CRYP, REL) with 39 open-source anchors into the database - Backend: POST/PUT/DELETE endpoints for canonical controls CRUD - Frontend proxy: PUT and DELETE methods added to canonical route - Frontend: Control Library with create/edit/delete UI, full form with open anchor management, scope, requirements, evidence, test procedures Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 00:28:21 +01:00
Benjamin Admin	453eec9ed8	fix: correct canonical control proxy paths to include /compliance prefix ... The backend mounts the compliance router at /api/compliance, so canonical control endpoints are at /api/compliance/v1/canonical/*, not /api/v1/canonical/*. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-12 20:49:06 +01:00
Benjamin Admin	050f353192	feat(canonical-controls): Canonical Control Library — rechtssichere Security Controls ... Eigenstaendig formulierte Security Controls mit unabhaengiger Taxonomie und Open-Source-Verankerung (OWASP, NIST, ENISA). Keine BSI-Nomenklatur. - Migration 044: 5 DB-Tabellen (frameworks, controls, sources, licenses, mappings) - 10 Seed Controls mit 39 Open-Source-Referenzen - License Gate: Quellen-Berechtigungspruefung (analysis/excerpt/embeddings/product) - Too-Close-Detektor: 5 Metriken (exact-phrase, token-overlap, ngram, embedding, LCS) - REST API: 8 Endpoints unter /v1/canonical/ - Go Loader mit Multi-Index (ID, domain, severity, framework) - Frontend: Control Library Browser + Provenance Wiki - CI/CD: validate-controls.py Job (schema, no-leak, open-anchors) - 67 Tests (8 Go + 59 Python), alle PASS - MkDocs Dokumentation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-12 19:55:06 +01:00
Benjamin Admin	8442115e7c	fix(rag): Fix bash compatibility + missing mkdir in phase functions ... - Replace ${var,,} (bash 4+) with $(echo | tr) for macOS bash 3.2 compat - Add mkdir -p to phase_gesetze, phase_eu, phase_templates, phase_datenschutz, phase_dach — prevents download failures when running phases individually Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-12 16:44:15 +01:00
Benjamin Admin	999cc81c78	feat(rag): Phase J — Security Guidelines & Standards (NIST, OWASP, ENISA) ... Add phase_security() with 15 documents across 3 sub-phases: - J1: 7 NIST standards (SP 800-53, 800-218, 800-63, 800-207, 8259A/B, AI RMF) - J2: 6 OWASP projects (Top 10, API Security, ASVS, MASVS, SAMM, Mobile Top 10) - J3: 2 ENISA guides (Procurement Hospitals, Cloud Security SMEs) All documents are commercially licensed (Public Domain / CC BY / CC BY-SA). Wire up 'security' phase in dispatcher and workflow yaml. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-12 16:14:44 +01:00
Benjamin Admin	ff66612beb	fix(rag): Make download failures non-fatal — prevent set -e from aborting entire ingestion ... download_pdf() and extract_gesetz_html() now return 0 on failure and clean up partial files. This prevents set -euo pipefail from aborting the entire script when a single download fails (e.g. EUR-Lex timeout, BSI redirect). Root cause of H2 EU loop only processing 1 document in Run #724: first failed download_pdf returned 1, triggering set -e script abort. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-12 15:56:23 +01:00
Benjamin Admin	42ec3cad6d	feat(rag): Phase I DACH-Erweiterung — Gesetze, Templates, Urteile ... New ingestion phase 'dach' adds missing documents from DACH catalog: I1: UStG (Retention), MStV (Impressum) I2: DSK Muster-VVT, DSK KP5 DSFA, BfDI Beispiel-VVT (DL-DE/BY-2.0) I3: BSI IT-Grundschutz Kompendium 2024 (CC BY-SA 4.0) I4: 7 Gerichtsentscheidungen as Praxisanker: - DE: LG Bonn 1&1, BGH Planet49, BGH Art.82 (2x) - AT: OGH Schutzzweck, OGH Art.15+82 EuGH-Vorlage - CH: BVGer DSG-Auskunft, BGer Datensperre Trigger: workflow_dispatch phase=dach Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-12 14:36:59 +01:00
Benjamin Admin	9945a62a50	fix(rag): docker cp into /workspace_scripts, then copy at runtime ... docker cp fails when target dir doesn't exist in a created container. Copy scripts to /workspace_scripts, then cp them at container start. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-12 14:24:36 +01:00
Benjamin Admin	eef1c2e7d3	fix(rag): Use docker cp to inject checked-out scripts ... The runner container can't access host paths directly, so the deploy dir scripts were always stale. Now uses docker create + docker cp + docker start to copy the freshly checked-out scripts into the ingestion container before starting it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-12 13:37:57 +01:00
Benjamin Admin	a0e2a35e66	fix(rag): Git pull deploy dir before ingestion ... The RAG workflow mounts scripts from /opt/breakpilot-compliance/scripts (deploy dir) but this may not have the latest fixes if CI hasn't deployed yet. Add explicit git pull before running ingestion. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-12 13:13:33 +01:00
Benjamin Admin	57f390190d	fix(rag): Arithmetic error, dedup auth, EGBGB timeout ... - collection_count() returns 0 (not ?) on failure — fixes arithmetic error - Pass QDRANT_API_KEY to ingestion container for dedup checks - Include api-key header in collection_count() and dedup scroll queries - Lower large-file threshold to 256KB (EGBGB 310KB was timing out) - More targeted EGBGB XML extraction (Art. 246a + Anlage only) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-12 12:05:07 +01:00
Benjamin Admin	cf60c39658	fix(scope-engine): Normalize UPPERCASE trigger docs to lowercase ScopeDocumentType ... Critical bug fix: mandatoryDocuments in Hard-Trigger-Rules used UPPERCASE names (VVT, TOM, DSE) that never matched lowercase ScopeDocumentType keys (vvt, tom, dsi). This meant no trigger documents were ever recognized as mandatory in buildDocumentScope(). - Add normalizeDocType() mapping function with alias support (DSE→dsi, LOESCHKONZEPT→lf, DSR_PROZESS→betroffenenrechte, etc.) - Fix buildDocumentScope() to use normalized doc types - Fix estimateEffort() to use lowercase keys matching ScopeDocumentType - Add 2 tests for UPPERCASE normalization and alias resolution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-12 09:39:31 +01:00
Benjamin Admin	c88653b221	fix(rag): Dedup check, BGB split, GewO timeout, arithmetic fix ... - Add Qdrant dedup check in upload_file() — skip if regulation_id already exists - Split BGB (2.7MB) into 5 targeted parts via XML extraction: AGB §§305-310, Fernabsatz §§312-312k, Kaufrecht §§433-480, Widerruf §§355-361, Digitale Produkte §§327-327u - Lower large-file threshold 512KB→384KB (fixes GewO 432KB timeout) - Fix arithmetic syntax error when collection_count returns "?" - Replace EGBGB PDF (was empty) with XML extraction - Add unzip to Alpine container for XML archives Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-12 09:39:09 +01:00
Benjamin Admin	87d06c8b20	fix(rag): Handle large file uploads + don't abort on individual failures ... - Extended timeout (15 min) for files > 500KB (BGB is 1.5MB) - upload_file returns 0 even on failure so set -e doesn't kill script - Failed uploads are still counted and reported in summary Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-11 23:33:28 +01:00
Benjamin Admin	0b47612272	fix(rag): Always run download phase before ingestion phases ... The gesetze phase failed because it expects text files created by the download phase. Now the workflow automatically runs download first for any phase that depends on it. Also adds git and python3 to the alpine container for repo cloning and text extraction. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-11 23:13:33 +01:00
Benjamin Admin	c14b31b3bc	fix(docker): Ensure public dir exists in Next.js builds + Hetzner compose fixes ... - admin-compliance/Dockerfile: mkdir -p public before build - developer-portal/Dockerfile: mkdir -p public before build (fixes "failed to calculate checksum /app/public: not found") - docker-compose.hetzner.yml: Override core-health-check to exit immediately (Core doesn't run on Hetzner) - Network override: external:false (auto-create breakpilot-network) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-11 19:11:26 +01:00
Benjamin Admin	0b836f7e2d	fix(ci): Run docker compose from helper container with deploy dir mounted ... The runner container has Docker socket but no host filesystem access. docker compose needs to read YAML files, so run build+deploy inside a helper container that has both Docker socket and the deploy dir mounted. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-11 18:31:19 +01:00
Benjamin Admin	18d9eec654	fix(ci): Use --entrypoint sh for alpine/git (default entrypoint is git) ... Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-11 18:14:58 +01:00
Benjamin Admin	339505feed	fix(ci): Fix Hetzner deploy — host filesystem access + network + dependencies ... Problems fixed: 1. Deploy step couldn't access /opt/breakpilot-compliance (host path not mounted in runner container). Now uses alpine/git helper container with host bind-mount for git ops, then docker compose with host paths. 2. breakpilot-network was external:true but Core doesn't run on Hetzner. Override in hetzner.yml creates the network automatically. 3. core-health-check blocks startup waiting for Core. Override in hetzner.yml makes it exit immediately. 4. RAG ingestion script now respects RAG_URL/QDRANT_URL env vars. 5. RAG workflow discovers network dynamically from running containers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-11 18:11:05 +01:00
Benjamin Admin	23b9808bf3	debug(ci): Discovery step to find RAG service on Hetzner ... Temporary commit to discover Docker container names and networks on Hetzner, since breakpilot-network doesn't exist there. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-11 17:58:46 +01:00
Benjamin Admin	c3654bc9ea	fix(ci): Spawn ingestion container on breakpilot-network ... Instead of trying to connect the runner to breakpilot-network, spawn a new alpine container directly on it via docker run. Added debug output for network/container visibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-11 17:53:06 +01:00