121 Commits

Author	SHA1	Message	Date
Benjamin Admin	200facda6a	fix: use CAST(:dd AS jsonb) instead of :dd::jsonb in _write_review ... SQLAlchemy's text() parser doesn't properly handle :param::type syntax — it fails to recognize :dd as a bind parameter when followed by ::jsonb. Using CAST(:dd AS jsonb) instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-24 08:48:58 +01:00
Benjamin Admin	9282850138	fix: add db.rollback() to batch dedup error handlers ... SQLAlchemy sessions enter a failed state after SQL errors. Without rollback(), all subsequent queries on the same session fail with InFailedSqlTransaction. Added try/except with rollback in _mark_duplicate, _mark_duplicate_to, _write_review, cross-group pass, and the main phase1 loop. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-24 08:41:36 +01:00
Benjamin Admin	770f0b5ab0	fix: adapt batch dedup to NULL pattern_id — group by merge_group_hint ... All Pass 0b controls have pattern_id=NULL. Rewritten to: - Phase 1: Group by merge_group_hint (action:object:trigger), 52k groups - Phase 2: Cross-group embedding search for semantically similar masters - Qdrant search uses unfiltered cross-regulation endpoint - API param changed: pattern_id → hint_filter Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-24 07:24:02 +01:00
Benjamin Admin	35784c35eb	feat: Batch Dedup Runner — 85k→~18-25k Master Controls ... Adds batch orchestration for deduplicating ~85k Pass 0b atomic controls into ~18-25k unique masters with M:N parent linking. New files: - migrations/078_batch_dedup.sql: merged_into_uuid column, perf indexes, link_type CHECK extended for cross_regulation - batch_dedup_runner.py: BatchDedupRunner with quality scoring, merge-hint grouping, title-identical short-circuit, parent-link transfer, and cross-regulation pass - tests/test_batch_dedup_runner.py: 21 tests (all passing) Modified: - control_dedup.py: optional collection param on Qdrant functions - crosswalk_routes.py: POST/GET batch-dedup endpoints Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-24 07:06:38 +01:00
Benjamin Admin	cce2707c03	fix: update 61 outdated test mocks to match current schemas ... Tests were failing due to stale mock objects after schema extensions: - DSFA: add _mapping property to _DictRow, use proper mock instead of MagicMock - Company Profile: add 6 missing fields (project_id, offering_urls, etc.) - Legal Templates/Policy: update document type count 52→58 - VVT: add 13 missing attributes to activity mock - Legal Documents: align consent test assertions with production behavior Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-24 06:40:42 +01:00
Benjamin Admin	e6201d5239	feat: Anti-Fake-Evidence System (Phase 1-4b) ... Implement full evidence integrity pipeline to prevent compliance theater: - Confidence levels (E0-E4), truth status tracking, assertion engine - Four-Eyes approval workflow, audit trail, reject endpoint - Evidence distribution dashboard, LLM audit routes - Traceability matrix (backend endpoint + Compliance Hub UI tab) - Anti-fake badges, control status machine, normative patterns - 2 migrations, 4 test suites, MkDocs documentation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-23 17:15:45 +01:00
Benjamin Admin	48ca0a6bef	feat: Framework Decomposition Engine + Composite Detection for Pass 0b ... Adds a routing layer between Pass 0a and Pass 0b that classifies obligations into atomic/compound/framework_container. Framework-container obligations (e.g. "CCM-Praktiken fuer AIS") are decomposed into concrete sub-obligations via an internal framework registry before Pass 0b composition. - New: framework_decomposition.py with routing, matching, decomposition - New: Framework registry (NIST SP 800-53, OWASP ASVS, CSA CCM) as JSON - New: Composite detection flags on atomic controls (is_composite, atomicity) - New: gen_meta fields: framework_ref, framework_domain, decomposition_source - Integration: _route_and_compose() in run_pass0b() deterministic path - 248 tests (198 decomposition + 50 framework), all passing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-23 12:11:55 +01:00
Benjamin Admin	1a63f5857b	feat: Deterministic Control Composition Engine v2 for Pass 0b ... Replace LLM-based Pass 0b with rule-based deterministic engine that composes atomic controls from obligation data without any LLM call. Engine features: - 24 action types (implement, configure, define, document, maintain, review, monitor, assess, audit, test, verify, validate, report, notify, train, restrict_access, encrypt, delete, retain, ensure, approve, remediate, perform, obtain) - 19 object classes (policy, procedure, register, record, report, technical_control, access_control, cryptographic_control, configuration, account, system, data, interface, role, training, incident, risk_artifact, process, consent) - Compound action splitting with no-split phrases - Title pattern: "{Object} {state_suffix}" (24 state suffixes) - Statement field: "{condition} {object} ist {trigger} {action}" - Pattern candidates for downstream categorization (26 specific combos + 24 action fallbacks) - Structured timing: deadline_hours + frequency extraction - Confidence scoring (0.3 base + action/object/trigger/template) - Merge group hints for dedup: "{action}:{norm_obj}:{trigger}" - Synonym-based object normalization (50+ German synonyms) - 16 specific (action_type, object_class) template overrides - Output validator with Pflichtfelder + Negativregeln + Warnregeln - All new fields serialized into gen_meta JSONB (no migration needed) Tests: 185 passed (33 new tests covering all engine components) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-23 11:05:48 +01:00
Benjamin Admin	649a3c5e4e	perf: switch Pass 0b default model to Haiku 4.5 ... Benchmark shows Haiku is 2.5x faster than Sonnet at 5x lower cost for this JSON structuring task. Quality is equivalent. $142 vs $705 for 75K obligations, ~2.8 days vs ~7 days. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-23 09:12:01 +01:00
Benjamin Admin	bdd2f6fa0f	fix: cap Anthropic max_tokens to 16384 for Pass 0b batches ... Previous formula (batch_size * 1500) exceeded Claude's 16K output limit for batch_size > 10, causing API failures and Ollama fallback. New formula: min(16384, max(4096, batch_size * 500)) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-23 08:50:45 +01:00
Benjamin Admin	ac6134ce6d	feat: control_parent_links population + traceability API + frontend ... - _write_atomic_control() now uses RETURNING id and inserts into control_parent_links (M:N) with source_regulation, source_article, and obligation_candidate_id parsed from parent's source_citation - New _parse_citation() helper for JSONB source_citation extraction - New GET /controls/{id}/traceability endpoint returning full chain: parent links with obligations, child controls, source_count - Backend: control_type filter (atomic/rich) for controls + count - Frontend: Rechtsgrundlagen section in ControlDetail showing all parent links per source regulation with obligation text + strength - Frontend: Atomic/Rich filter dropdown in Control Library list - Frontend: GenerationStrategyBadge recognizes 'pass0b' strategy - Tests: 3 new tests for parent_link creation + citation parsing, existing batch test mock updated for RETURNING clause Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-23 08:14:29 +01:00
Benjamin Admin	a14e2f3a00	feat(decomposition): add merge pass, enrichment, and Pass 0b refinements ... Add obligation refinement pipeline between Pass 0a and 0b: - Merge pass: rule-based dedup of implementation-level duplicate obligations within the same parent control (Jaccard similarity on action+object) - Enrich pass: classify trigger_type (event/periodic/continuous) and detect is_implementation_specific from obligation text (regex-based, no LLM) - Pass 0b: skip merged obligations, cap severity for impl-specific, override category to 'testing' for test obligations - Migration 075: merged_into_id, trigger_type, is_implementation_specific - Two new API endpoints: merge-obligations, enrich-obligations - 30+ new tests (122 total, all passing) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-21 22:27:09 +01:00
Benjamin Admin	71b8c33270	fix(docker): make torch/sentence-transformers optional to unblock builds ... Move torch and sentence-transformers to requirements-reranker.txt so the main Docker build succeeds even if these large packages fail to install. The reranker code already handles missing imports gracefully when RERANK_ENABLED=false (the default). This fixes the production deployment — builds were failing because of the ~800MB torch dependency, preventing ALL new code from deploying. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-21 15:06:51 +01:00
Benjamin Admin	f2924a58ed	debug: add /debug/routers endpoint to diagnose import failures ... Crosswalk routes returning 404 on production. This adds a diagnostic endpoint that reports which sub-routers failed to load and why. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-21 12:23:26 +01:00
Benjamin Admin	643b26618f	feat: Control Library UI, dedup migration, QA tooling, docs ... - Control Library: parent control display, ObligationTypeBadge, GenerationStrategyBadge variants, evidence string fallback - API: expose parent_control_uuid/id/title in canonical controls - Fix: DSFA SQLAlchemy 2.0 Row._mapping compatibility - Migration 074: control_parent_links + control_dedup_reviews tables - QA scripts: benchmark, gap analysis, OSCAL import, OWASP cleanup, phase5 normalize, phase74 gap fill, sync_db, run_job - Docs: dedup engine, RAG benchmark, lessons learned, pipeline docs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-21 11:56:08 +01:00
Benjamin Admin	c52dbdb8f1	feat(rag): optimize RAG pipeline — JSON-Mode, CoT, Hybrid Search, Re-Ranking, Cross-Reg Dedup, chunk 1024 ... Phase 1 (LLM Quality): - Add format=json to all Ollama payloads (obligation_extractor, control_generator, citation_backfill) - Add Chain-of-Thought analysis steps to Pass 0a/0b system prompts Phase 2 (Retrieval Quality): - Hybrid search via Qdrant Query API with RRF fusion + automatic text index (legal_rag.go) - Fallback to dense-only search if Query API unavailable - Cross-encoder re-ranking with BGE Reranker v2 (RERANK_ENABLED=false by default) - CPU-only PyTorch dependency to keep Docker image small Phase 3 (Data Layer): - Cross-regulation dedup pass (threshold 0.95) links controls across regulations - DedupResult.link_type field distinguishes dedup_merge vs cross_regulation - Chunk size defaults updated 512/50 → 1024/128 for new ingestions only - Existing collections and controls are NOT affected Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-21 11:49:43 +01:00
Benjamin Admin	1cc34c23d9	feat(document-generator): 33 policy + module document templates ... - Migration 071: 14 IT-Security policy templates (ISO 27001/BSI) information_security, access_control, password, encryption, logging, backup, incident_response, change_management, patch_management, asset_management, cloud_security, devsecops, secrets_management, vulnerability_management - Migration 072: 15 Data/HR/Vendor/BCM policy templates data_protection, data_classification, data_retention, data_transfer, privacy_incident, employee_security, security_awareness, remote_work, offboarding, vendor_risk_management, third_party_security, supplier_security, business_continuity, disaster_recovery, crisis_management - Migration 073: 4 module document reference templates vvt_register, tom_documentation, loeschkonzept, pflichtenregister - TemplateType union: 17 → 61 types with German labels - VALID_DOCUMENT_TYPES: +6 types (cybersecurity_policy, dsfa, 4 module docs) - CATEGORIES: new "DSGVO-Dokumente" category for module documents Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-19 23:27:25 +01:00
Benjamin Admin	5dd7a27336	fix(pipeline): add missing regulation codes to LICENSE_MAP ... eu_2023_1542 (Batterieverordnung), eu_2023_988 (GPSR), nist_sp800_218, nist_privacy_1_0, owasp_mobile_top10 were defaulting to Rule 3 (restricted) instead of their correct rules. This caused 68/71 controls to be flagged as too_close in the last pipeline run. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-19 22:14:10 +01:00
Benjamin Admin	c3afa628ed	feat(sdk): vendor-compliance cross-module integration — VVT, obligations, TOM, loeschfristen ... Integrate the vendor-compliance module with four DSGVO modules to eliminate data silos and resolve the VVT processor tab's ephemeral state problem. - Reposition vendor-compliance sidebar from seq 4200 to 2500 (after VVT) - VVT: replace ephemeral ProcessorRecord state with Vendor-API fetch (read-only) - Obligations: add linked_vendor_ids (JSONB) + compliance check #12 MISSING_VENDOR_LINK - TOM: add vendor TOM-controls cross-reference table in overview tab - Loeschfristen: add linked_vendor_ids (JSONB) + vendor picker + document section - Migrations: 069_obligations_vendor_link.sql, 070_loeschfristen_vendor_link.sql - Tests: 12 new backend tests (125 total pass) - Docs: update obligations.md + vendors.md with cross-module integration Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-19 13:59:43 +01:00
Benjamin Admin	4b1eede45b	feat(tom): audit document, compliance checks, 25 controls, canonical control mapping ... Phase A: TOM document HTML generator (12 sections, inline CSS, A4 print) Phase B: TOMDocumentTab component (org-header form, revisions, print/download) Phase C: 11 compliance checks with severity-weighted scoring Phase D: MkDocs documentation for TOM module Phase E: 25 new controls (63 → 88) in 13 categories Canonical Control Mapping (three-layer architecture): - Migration 068: tom_control_mappings + tom_control_sync_state tables - 6 API endpoints: sync, list, by-tom, stats, manual add, delete - Category mapping: 13 TOM categories → 17 canonical categories - Frontend: sync button + coverage card (Overview), drill-down (Editor), belegende Controls count (Document) - 20 tests (unit + API with mocked DB) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-19 11:56:53 +01:00
Benjamin Admin	2a70441eaa	feat(sdk): VVT master libraries, process templates, Loeschfristen profiling + document ... VVT: Master library tables (7 catalogs), 500+ seed entries, process templates with instantiation, library API endpoints + 18 tests. Loeschfristen: Baseline catalog, compliance checks, profiling engine, HTML document generator, MkDocs documentation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-19 11:56:25 +01:00
Benjamin Admin	f2819b99af	feat(pipeline): v3 — scoped control applicability + source_type classification ... Phase 4: source_type (law/guideline/standard/restricted) on source_citation - NIST/OWASP/ENISA correctly shown as "Standard" instead of "Gesetzliche Grundlage" - Dynamic frontend labels based on source_type - Backfill endpoint POST /v1/canonical/generate/backfill-source-type Phase v3: Scoped Control Applicability - 3 new fields: applicable_industries, applicable_company_size, scope_conditions - LLM prompt extended with 39 industries, 5 company sizes, 10 scope signals - All 5 generation paths (Rule 1/2/3, batch structure, batch reform) updated - _build_control_from_json: parsing + validation (string→list, size validation) - _store_control: writes 3 new JSONB columns - API: response models, create/update requests, SELECT queries extended - Migration 063: 3 new JSONB columns with GIN indexes - 110 generator tests + 28 route tests = 138 total, all passing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-18 16:28:05 +01:00
Benjamin Admin	148c7ba3af	feat(qa): recital detection, review split, duplicate comparison ... Add _detect_recital() to QA pipeline — flags controls where source_original_text contains Erwägungsgrund markers instead of article text (28% of controls with source text affected). - Recital detection via regex + phrase matching in QA validation - 10 new tests (TestRecitalDetection), 81 total - ReviewCompare component for side-by-side duplicate comparison - Review mode split: Duplikat-Verdacht vs Rule-3-ohne-Anchor tabs - MkDocs: recital detection documentation - Detection script for bulk analysis (scripts/find_recital_controls.py) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-18 08:20:02 +01:00
Benjamin Admin	a9e0869205	feat(pipeline): pipeline_version v2, migration 062, docs + 71 tests ... - Add PIPELINE_VERSION=2 constant and pipeline_version column to canonical_controls and canonical_processed_chunks (migration 062) - Anthropic API decides chunk relevance via null-returns (skip_prefilter) - Annex/appendix chunks explicitly protected in prompts - Fix 6 failing tests (CRYP domain, _process_batch tuple return) - Add TestPipelineVersion + TestRegulationFilter test classes (10 new tests) - Add MkDocs page: control-generator-pipeline.md (541 lines) - Update canonical-control-library.md with v2 pipeline diagram - Update testing.md with 71-test breakdown table Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-17 17:31:11 +01:00
Benjamin Admin	653aad57e3	Let Anthropic API decide chunk relevance instead of local prefilter ... Updated both structure_batch and reformulate_batch prompts to return null for chunks without actionable requirements (definitions, TOCs, scope-only). Explicit instruction to always process annexes/appendices as they often contain concrete technical requirements. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-17 16:44:01 +01:00
Benjamin Admin	a7f7e57dd7	Add skip_prefilter option to control generator ... Local LLM prefilter (llama3.2 3B) was incorrectly skipping annex chunks that contain concrete requirements. Added skip_prefilter flag to bypass the local pre-filter and send all chunks directly to Anthropic API. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-17 16:30:57 +01:00
Benjamin Admin	567e82ddf5	Fix stale DB session after long embedding pre-load ... The embedding pre-load for 4998 existing controls takes ~16 minutes, causing the SQLAlchemy session to become invalid. Added rollback after pre-load completes to reset the session before subsequent DB operations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-17 14:34:44 +01:00
Benjamin Admin	36ef34169a	Fix regulation_filter bypass for chunks without regulation_code ... Chunks without a regulation_code were silently passing through the filter in _scan_rag(), causing unrelated documents (e.g. Data Act, legal templates) to be included in filtered generation jobs. Now chunks without reg_code are skipped when regulation_filter is active. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-17 13:38:25 +01:00
Benjamin Admin	d22c47c9eb	feat(pipeline): Anthropic Batch API, source/regulation filter, cost optimization ... - Add Anthropic API support to decomposition Pass 0a/0b (prompt caching, content batching) - Add Anthropic Batch API (50% cost reduction, async 24h processing) - Add source_filter (ILIKE on source_citation) for regulation-based filtering - Add category_filter to Pass 0a for selective decomposition - Add regulation_filter to control_generator for RAG scan phase filtering (prefix match on regulation_code — enables CE + Code Review focus) - New API endpoints: batch-submit-0a, batch-submit-0b, batch-status, batch-process - 83 new tests (all passing) Cost reduction: $2,525 → ~$600-700 with all optimizations combined. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-17 13:22:01 +01:00
Benjamin Admin	825e070ed9	feat(multi-layer): complete Multi-Layer Control Architecture (Phases 1-8 + Pass 0) ... Implements the full Multi-Layer Control Architecture for migrating ~25,000 Rich Controls into atomic, deduplicated Master Controls with full traceability. Architecture: Legal Source → Obligation → Control Pattern → Master Control → Customer Instance New services: - ObligationExtractor: 3-tier extraction (exact → embedding → LLM) - PatternMatcher: 2-tier matching (keyword + embedding + domain-bonus) - ControlComposer: Pattern + Obligation → Master Control - PipelineAdapter: Pipeline integration + Migration Passes 1-5 - DecompositionPass: Pass 0a/0b — Rich Control → atomic Controls - CrosswalkRoutes: 15 API endpoints under /v1/canonical/ New DB schema: - Migration 060: obligation_extractions, control_patterns, crosswalk_matrix - Migration 061: obligation_candidates, parent_control_uuid tracking Pattern Library: 50 YAML patterns (30 core + 20 IT-security) Go SDK: Pattern loader with YAML validation and indexing Documentation: MkDocs updated with full architecture overview 500 Python tests passing across all components. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-17 09:00:37 +01:00
Benjamin Admin	4f6bc8f6f6	feat(training+controls): interactive video pipeline, training blocks, control generator, CE libraries ... Interactive Training Videos (CP-TRAIN): - DB migration 022: training_checkpoints + checkpoint_progress tables - NarratorScript generation via Anthropic (AI Teacher persona, German) - TTS batch synthesis + interactive video pipeline (slides + checkpoint slides + FFmpeg) - 4 new API endpoints: generate-interactive, interactive-manifest, checkpoint submit, checkpoint progress - InteractiveVideoPlayer component (HTML5 Video, quiz overlay, seek protection, progress tracking) - Learner portal integration with automatic completion on all checkpoints passed - 30 new tests (handler validation + grading logic + manifest/progress + seek protection) Training Blocks: - Block generator, block store, block config CRUD + preview/generate endpoints - Migration 021: training_blocks schema Control Generator + Canonical Library: - Control generator routes + service enhancements - Canonical control library helpers, sidebar entry - Citation backfill service + tests - CE libraries data (hazard, protection, evidence, lifecycle, components) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-16 21:41:48 +01:00
Benjamin Admin	3b2006ebce	feat(iace): add hazard-matching-engine with component library, tag system, and pattern engine ... Implements Phases 1-4 of the IACE Hazard-Matching-Engine: - 120 machine components (C001-C120) in 11 categories - 20 energy sources (EN01-EN20) - ~85 tag taxonomy across 5 domains - 44 hazard patterns with AND/NOT matching logic - Pattern engine with tag resolution and confidence scoring - 8 new API endpoints (component-library, energy-sources, tags, patterns, match/apply) - Completeness gate G09 for pattern matching - 320 tests passing (36 new) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-16 08:50:11 +01:00
Benjamin Admin	c8fd9cc780	feat(control-library): document-grouped batching, generation strategy tracking, sort by source ... - Group chunks by regulation_code before batching for better LLM context - Add generation_strategy column (ungrouped=v1, document_grouped=v2) - Add v1/v2 badge to control cards in frontend - Add sort-by-source option with visual group headers - Add frontend page tests (18 tests) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-15 15:10:52 +01:00
Benjamin Admin	dd09fa7a46	feat: CRA wiki, cybersecurity policy template, Phase H RAG ingestion ... - Wiki: add CRA category with 3 articles (Grundlagen, 35 Security Controls, CRA+NIS2+AI Act Framework) - Document Generator: add CRA-konforme Cybersecurity Policy template with 21 sections covering governance, SSDLC, vulnerability management, incident response (24h/72h), SBOM, patch management - RAG: ingest Phase H — 17 EU regulations + 2 NIST frameworks now in Qdrant (CRA, AI Act, NIS2, DSGVO, DMA, GPSR, Batterieverordnung, etc.) - Phase H script: add scripts/ingest-phase-h.sh for reproducible ingestion - rag-sources.md: update status to ingestiert, add CRA entry Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-15 00:43:46 +01:00
Benjamin Admin	f3e05c1bf7	feat: enhance whistleblower HinSchG content, fix control-library filter layout ... - Whistleblower page: expand overview tab with comprehensive HinSchG legal info (Gesetzliche Grundlage, Fristen-Cards, Anwendungsbereich, Schutz des Hinweisgebers) - StepHeader: enrich whistleblower tips with detailed HinSchG paragraphs and sanctions - Wiki: add migration 054 with 5 new/updated HinSchG articles (Anwendungsbereich, Hinweisgeberschutz, Meldestellen, Verfahrensablauf, Datenschutz-Anforderungen) - MKDocs: rewrite whistleblower docs with full legal basis, architecture, API, DB schema - Control library: fix filter dropdown overflow by splitting into search + filter rows Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-15 00:23:19 +01:00
Benjamin Admin	2ed1c08acf	feat: enhance legal basis display, add batch processing tests and docs ... - Backfill 81 controls with empty source_citation.source from generation_metadata - Add fallback to generation_metadata.source_regulation in ControlDetail blue box - Improve Rule 3 amber box text for reformulated controls - Add 30 new tests for batch processing (TestParseJsonArray, TestBatchSizeConfig, TestBatchProcessingLoop) — all 61 control generator tests passing - Fix stale test_config_defaults assertion (max_controls 50→0) - Update canonical-control-library.md with batch processing pipeline docs, processed chunks tracking, migration guide, and stats endpoint - Update testing.md with canonical control generator test section Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-14 23:51:52 +01:00
Benjamin Admin	a9f291ff49	test+docs: add policy library tests (67 tests) and MKDocs documentation ... - New test_policy_templates.py: 67 tests covering all 29 policy types, API creation, filtering, placeholders, seed script validation - Updated test_legal_template_routes.py: fix type count 16→52 - New MKDocs page policy-bibliothek.md with full template reference - Updated dokumentengenerierung.md and rechtliche-texte.md with cross-refs - Added policy-bibliothek to mkdocs.yml navigation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-14 22:50:50 +01:00
Benjamin Admin	0171d611f6	feat: add policy library with 29 German policy templates ... Add 29 new document types (IT security, data, personnel, vendor, BCM policies) to VALID_DOCUMENT_TYPES and 5 category pills to the document generator UI. Include seed script for production DB population. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-14 22:37:33 +01:00
Benjamin Admin	637fab6fdb	fix: migration runner strips BEGIN/COMMIT and guards missing tables ... Root cause: migrations 046-047 used explicit BEGIN/COMMIT which conflicts with psycopg2 implicit transactions, and ALTER TABLE on canonical_controls fails when the table doesn't exist on production. This blocked all subsequent migrations (048-053). Changes: - migration_runner.py: strip BEGIN/COMMIT from SQL before executing - 046: wrap canonical_controls ALTER in DO $$ IF EXISTS block - 047: wrap canonical_controls ALTER in DO $$ IF EXISTS block Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-14 21:59:10 +01:00
Benjamin Admin	d462141ccd	fix: migration runner continues on failure instead of aborting ... Previously, a single failed migration would abort all subsequent migrations via raise RuntimeError. Now the runner logs the failure and continues with remaining migrations, so independent schema changes (e.g. 050-053) are not blocked by an unrelated failure in an earlier migration (e.g. 048). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-14 21:54:08 +01:00
Benjamin Admin	5f8aebf5b1	fix: make migrations 048/049 safe for environments without canonical tables ... Migrations 048 and 049 reference canonical_processed_chunks and canonical_controls tables which may not exist on all environments. Wrap ALTER TABLE statements in DO blocks that check for table existence first. This unblocks migrations 050-053 on production. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-14 21:45:00 +01:00
Benjamin Admin	49ce417428	feat: add compliance modules 2-5 (dashboard, security templates, process manager, evidence collector) ... Module 2: Extended Compliance Dashboard with roadmap, module-status, next-actions, snapshots, score-history Module 3: 7 German security document templates (IT-Sicherheitskonzept, Datenschutz, Backup, Logging, Incident-Response, Zugriff, Risikomanagement) Module 4: Compliance Process Manager with CRUD, complete/skip/seed, ~50 seed tasks, 3-tab UI Module 5: Evidence Collector Extended with automated checks, control-mapping, coverage report, 4-tab UI Also includes: canonical control library enhancements (verification method, categories, dedup), control generator improvements, RAG client extensions 52 tests pass, frontend builds clean. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-14 21:03:04 +01:00
Benjamin Admin	13d13c8226	fix: add all RAG regulation codes to license mapping ... Many regulation codes (nist_sp800_53r5, eucsa, owasp_top10_2021, EDPB guidelines, EU laws, AT/FR/ES/NL/IT/HU laws) were defaulting to Rule 3 (restricted) because they weren't in REGULATION_LICENSE_MAP. Now all ~100 regulation codes from RAG are properly mapped to Rule 1 or 2. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-14 08:38:31 +01:00
Benjamin Admin	b6e6ffaaee	feat: add verification method, categories, and dedup UI to control library ... - Migration 047: verification_method + category columns, 17 category lookup table - Backend: new filters, GET /categories, GET /controls/{id}/similar (embedding-based) - Frontend: filter dropdowns, badges, dedup UI in ControlDetail with merge workflow - ControlForm: verification method + category selects - Provenance: verification methods, categories, master library strategy sections - Fix UUID cast syntax in generator routes (::uuid -> CAST) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-14 07:55:22 +01:00
Benjamin Admin	9812ff46f3	feat: add 7-stage control generator pipeline with 3 license rules ... - control_generator.py: RAG→License→Structure/Reform→Harmonize→Anchor→Store→Mark pipeline with Anthropic Claude API (primary) + Ollama fallback for LLM reformulation - anchor_finder.py: RAG-based + DuckDuckGo anchor search for open references - control_generator_routes.py: REST API for generate, job status, review queue, processed stats - 046_control_generator.sql: job tracking, chunk tracking, blocked sources tables; extends canonical_controls with license_rule, source_original_text, source_citation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 18:42:40 +01:00
Sharang Parnerkar	f1710fdb9e	fix: migrate deployment from Hetzner to Coolify (#1) ... ## Summary - Add Coolify deployment configuration (docker-compose, healthchecks, network setup) - Replace deploy-hetzner CI job with Coolify webhook deploy - Externalize postgres, qdrant, S3 for Coolify environment ## All changes since branch creation - Coolify docker-compose with Traefik labels and healthchecks - CI pipeline: deploy-hetzner → deploy-coolify (simple webhook curl) - SQLAlchemy 2.x text() compatibility fixes - Alpine-compatible Dockerfile fixes Co-authored-by: Sharang Parnerkar <parnerkarsharang@gmail.com> Reviewed-on: #1	2026-03-13 10:45:35 +00:00
Benjamin Admin	f738ca8c52	fix: make compliance router imports resilient to individual module failures ... Replaced bare imports with safe_import_router pattern — if one sub-router fails to import (e.g. missing dependency), other routers still load. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 09:46:52 +01:00
Benjamin Admin	c530898963	fix: replace Python 3.10+ union type syntax with typing.Optional for Pydantic v2 compat ... from __future__ import annotations breaks Pydantic BaseModel runtime type evaluation. Replaced str | None → Optional[str], list[str] → List[str] etc. in control_generator.py, anchor_finder.py, control_generator_routes.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 09:36:14 +01:00
Benjamin Admin	cdafc4d9f4	feat: auto-run SQL migrations on backend startup ... Adds migration_runner.py that executes pending migrations from migrations/ directory when backend-compliance starts. Tracks applied migrations in _migration_history table. Handles existing databases: detects if tables from migrations 001-045 already exist and seeds the history table accordingly, so only new migrations (046+) are applied. Skippable via SKIP_MIGRATIONS=true env var. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 09:14:18 +01:00
Benjamin Admin	de19ef0684	feat(control-generator): 7-stage pipeline for RAG→LLM→Controls generation ... Implements the Control Generator Pipeline that systematically generates canonical security controls from 150k+ RAG chunks across all compliance collections (BSI, NIST, OWASP, ENISA, EU laws, German laws). Three license rules enforced throughout: - Rule 1 (free_use): Laws/Public Domain — original text preserved - Rule 2 (citation_required): CC-BY/CC-BY-SA — text with citation - Rule 3 (restricted): BSI/ISO — full reformulation, no source traces New files: - Migration 046: job tracking, chunk tracking, blocked sources tables - control_generator.py: 7-stage pipeline (scan→classify→structure/reform→harmonize→anchor→store→mark) - anchor_finder.py: RAG + DuckDuckGo open-source reference search - control_generator_routes.py: REST API (generate, review, stats, blocked-sources) - test_control_generator.py: license mapping, rule enforcement, anchor filtering tests Modified: - __init__.py: register control_generator_router - route.ts: proxy generator/review/stats endpoints - page.tsx: Generator modal, stats panel, state filter, review queue, license badges Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-13 09:03:37 +01:00