Add "Dokumentenursprung" filter dropdown to the control library page.
Extracts unique source_citation.source values from controls, sorted by
frequency. Includes "Ohne Quelle" option for controls without source info.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Backfill 81 controls with empty source_citation.source from generation_metadata
- Add fallback to generation_metadata.source_regulation in ControlDetail blue box
- Improve Rule 3 amber box text for reformulated controls
- Add 30 new tests for batch processing (TestParseJsonArray, TestBatchSizeConfig,
TestBatchProcessingLoop) — all 61 control generator tests passing
- Fix stale test_config_defaults assertion (max_controls 50→0)
- Update canonical-control-library.md with batch processing pipeline docs,
processed chunks tracking, migration guide, and stats endpoint
- Update testing.md with canonical control generator test section
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New test_policy_templates.py: 67 tests covering all 29 policy types,
API creation, filtering, placeholders, seed script validation
- Updated test_legal_template_routes.py: fix type count 16→52
- New MKDocs page policy-bibliothek.md with full template reference
- Updated dokumentengenerierung.md and rechtliche-texte.md with cross-refs
- Added policy-bibliothek to mkdocs.yml navigation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 29 new document types (IT security, data, personnel, vendor, BCM
policies) to VALID_DOCUMENT_TYPES and 5 category pills to the document
generator UI. Include seed script for production DB population.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: migrations 046-047 used explicit BEGIN/COMMIT which
conflicts with psycopg2 implicit transactions, and ALTER TABLE
on canonical_controls fails when the table doesn't exist on
production. This blocked all subsequent migrations (048-053).
Changes:
- migration_runner.py: strip BEGIN/COMMIT from SQL before executing
- 046: wrap canonical_controls ALTER in DO $$ IF EXISTS block
- 047: wrap canonical_controls ALTER in DO $$ IF EXISTS block
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously, a single failed migration would abort all subsequent
migrations via raise RuntimeError. Now the runner logs the failure
and continues with remaining migrations, so independent schema
changes (e.g. 050-053) are not blocked by an unrelated failure
in an earlier migration (e.g. 048).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Migrations 048 and 049 reference canonical_processed_chunks and
canonical_controls tables which may not exist on all environments.
Wrap ALTER TABLE statements in DO blocks that check for table
existence first. This unblocks migrations 050-053 on production.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The frontend pages were calling /api/sdk/v1/compliance/process-tasks/*
and /api/sdk/v1/compliance/evidence-checks/* but no Next.js proxy
routes existed for these paths, causing 404s and empty data.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Many regulation codes (nist_sp800_53r5, eucsa, owasp_top10_2021, EDPB
guidelines, EU laws, AT/FR/ES/NL/IT/HU laws) were defaulting to Rule 3
(restricted) because they weren't in REGULATION_LICENSE_MAP. Now all
~100 regulation codes from RAG are properly mapped to Rule 1 or 2.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After every push to gitea, Claude now automatically polls health
endpoints and notifies the user when the deployment is ready for testing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace Hetzner references with Coolify. Deployment is now:
- Core + Compliance: Push gitea → Coolify auto-deploys
- Lehrer: stays local on Mac Mini
Updated: CLAUDE.md, MkDocs CI/CD pipeline, MkDocs index.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaced bare imports with safe_import_router pattern — if one sub-router
fails to import (e.g. missing dependency), other routers still load.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
from __future__ import annotations breaks Pydantic BaseModel runtime type
evaluation. Replaced str | None → Optional[str], list[str] → List[str] etc.
in control_generator.py, anchor_finder.py, control_generator_routes.py.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds migration_runner.py that executes pending migrations from
migrations/ directory when backend-compliance starts. Tracks applied
migrations in _migration_history table.
Handles existing databases: detects if tables from migrations 001-045
already exist and seeds the history table accordingly, so only new
migrations (046+) are applied.
Skippable via SKIP_MIGRATIONS=true env var.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Migration 045: Seed 10 controls (AUTH, NET, SUP, LOG, WEB, DATA, CRYP, REL)
with 39 open-source anchors into the database
- Backend: POST/PUT/DELETE endpoints for canonical controls CRUD
- Frontend proxy: PUT and DELETE methods added to canonical route
- Frontend: Control Library with create/edit/delete UI, full form with
open anchor management, scope, requirements, evidence, test procedures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The backend mounts the compliance router at /api/compliance, so canonical
control endpoints are at /api/compliance/v1/canonical/*, not /api/v1/canonical/*.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add phase_security() with 15 documents across 3 sub-phases:
- J1: 7 NIST standards (SP 800-53, 800-218, 800-63, 800-207, 8259A/B, AI RMF)
- J2: 6 OWASP projects (Top 10, API Security, ASVS, MASVS, SAMM, Mobile Top 10)
- J3: 2 ENISA guides (Procurement Hospitals, Cloud Security SMEs)
All documents are commercially licensed (Public Domain / CC BY / CC BY-SA).
Wire up 'security' phase in dispatcher and workflow yaml.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
download_pdf() and extract_gesetz_html() now return 0 on failure and clean up
partial files. This prevents set -euo pipefail from aborting the entire script
when a single download fails (e.g. EUR-Lex timeout, BSI redirect).
Root cause of H2 EU loop only processing 1 document in Run #724: first failed
download_pdf returned 1, triggering set -e script abort.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
docker cp fails when target dir doesn't exist in a created container.
Copy scripts to /workspace_scripts, then cp them at container start.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The runner container can't access host paths directly, so the
deploy dir scripts were always stale. Now uses docker create +
docker cp + docker start to copy the freshly checked-out scripts
into the ingestion container before starting it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The RAG workflow mounts scripts from /opt/breakpilot-compliance/scripts
(deploy dir) but this may not have the latest fixes if CI hasn't
deployed yet. Add explicit git pull before running ingestion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- collection_count() returns 0 (not ?) on failure — fixes arithmetic error
- Pass QDRANT_API_KEY to ingestion container for dedup checks
- Include api-key header in collection_count() and dedup scroll queries
- Lower large-file threshold to 256KB (EGBGB 310KB was timing out)
- More targeted EGBGB XML extraction (Art. 246a + Anlage only)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Critical bug fix: mandatoryDocuments in Hard-Trigger-Rules used UPPERCASE
names (VVT, TOM, DSE) that never matched lowercase ScopeDocumentType keys
(vvt, tom, dsi). This meant no trigger documents were ever recognized as
mandatory in buildDocumentScope().
- Add normalizeDocType() mapping function with alias support
(DSE→dsi, LOESCHKONZEPT→lf, DSR_PROZESS→betroffenenrechte, etc.)
- Fix buildDocumentScope() to use normalized doc types
- Fix estimateEffort() to use lowercase keys matching ScopeDocumentType
- Add 2 tests for UPPERCASE normalization and alias resolution
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Extended timeout (15 min) for files > 500KB (BGB is 1.5MB)
- upload_file returns 0 even on failure so set -e doesn't kill script
- Failed uploads are still counted and reported in summary
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The gesetze phase failed because it expects text files created by the
download phase. Now the workflow automatically runs download first for
any phase that depends on it. Also adds git and python3 to the alpine
container for repo cloning and text extraction.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- admin-compliance/Dockerfile: mkdir -p public before build
- developer-portal/Dockerfile: mkdir -p public before build
(fixes "failed to calculate checksum /app/public: not found")
- docker-compose.hetzner.yml: Override core-health-check to exit
immediately (Core doesn't run on Hetzner)
- Network override: external:false (auto-create breakpilot-network)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The runner container has Docker socket but no host filesystem access.
docker compose needs to read YAML files, so run build+deploy inside
a helper container that has both Docker socket and the deploy dir mounted.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Problems fixed:
1. Deploy step couldn't access /opt/breakpilot-compliance (host path not
mounted in runner container). Now uses alpine/git helper container with
host bind-mount for git ops, then docker compose with host paths.
2. breakpilot-network was external:true but Core doesn't run on Hetzner.
Override in hetzner.yml creates the network automatically.
3. core-health-check blocks startup waiting for Core. Override in
hetzner.yml makes it exit immediately.
4. RAG ingestion script now respects RAG_URL/QDRANT_URL env vars.
5. RAG workflow discovers network dynamically from running containers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Temporary commit to discover Docker container names and networks
on Hetzner, since breakpilot-network doesn't exist there.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Instead of trying to connect the runner to breakpilot-network,
spawn a new alpine container directly on it via docker run.
Added debug output for network/container visibility.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Join breakpilot-network so bp-core-rag-service is reachable
- Make RAG_URL/QDRANT_URL in script respect env vars (${VAR:-default})
- Remove complex fallback logic — fail fast if network not available
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The runner container doesn't always have /opt/breakpilot-compliance mounted.
Use the git-cloned workspace (current dir) and add multi-fallback for RAG API
URL (container network → localhost → host.docker.internal).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase H now includes:
- 16 German laws (PAngV, VSBG, ProdHaftG, BDSG, HGB, AO, DDG, TKG, etc.)
- 15 EUR-Lex EU laws (DSGVO, Consumer Rights Dir, Sale of Goods Dir,
E-Commerce Dir, Unfair Terms Dir, DMA, NIS2, Product Liability Dir, etc.)
- 2 NIST frameworks (CSF 2.0, Privacy Framework 1.0)
- 1 HLEG Ethics Guidelines
Updated rag-sources.md with complete inventory of already-ingested vs
new documents, plus Layer 2-5 TODO roadmap.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
