Benjamin Admin 35784c35eb feat: Batch Dedup Runner — 85k→~18-25k Master Controls ...

Adds batch orchestration for deduplicating ~85k Pass 0b atomic controls
into ~18-25k unique masters with M:N parent linking.

New files:
- migrations/078_batch_dedup.sql: merged_into_uuid column, perf indexes,
  link_type CHECK extended for cross_regulation
- batch_dedup_runner.py: BatchDedupRunner with quality scoring, merge-hint
  grouping, title-identical short-circuit, parent-link transfer, and
  cross-regulation pass
- tests/test_batch_dedup_runner.py: 21 tests (all passing)

Modified:
- control_dedup.py: optional collection param on Qdrant functions
- crosswalk_routes.py: POST/GET batch-dedup endpoints

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-24 07:06:38 +01:00

..

__init__.py

…

test_anti_fake_evidence_phase2.py

feat: Anti-Fake-Evidence System (Phase 1-4b)

2026-03-23 17:15:45 +01:00

test_anti_fake_evidence_phase3.py

feat: Anti-Fake-Evidence System (Phase 1-4b)

2026-03-23 17:15:45 +01:00

test_anti_fake_evidence_phase4.py

feat: Anti-Fake-Evidence System (Phase 1-4b)

2026-03-23 17:15:45 +01:00

test_anti_fake_evidence.py

feat: Anti-Fake-Evidence System (Phase 1-4b)

2026-03-23 17:15:45 +01:00

test_banner_routes.py

feat: Consent-Service Module nach Compliance migriert (DSR, E-Mail-Templates, Legal Docs, Banner)

2026-03-05 00:36:24 +01:00

test_batch_dedup_runner.py

feat: Batch Dedup Runner — 85k→~18-25k Master Controls

2026-03-24 07:06:38 +01:00

test_canonical_control_routes.py

feat: Control Library UI, dedup migration, QA tooling, docs

2026-03-21 11:56:08 +01:00

test_change_request_routes.py

feat(sdk): Multi-Tenancy, Versionierung, Change-Requests, Dokumentengenerierung (Phase 1-6)

2026-03-07 14:12:34 +01:00

test_citation_backfill.py

feat(rag): optimize RAG pipeline — JSON-Mode, CoT, Hybrid Search, Re-Ranking, Cross-Reg Dedup, chunk 1024

2026-03-21 11:49:43 +01:00

test_company_profile_extend.py

fix: update 61 outdated test mocks to match current schemas

2026-03-24 06:40:42 +01:00

test_company_profile_routes.py

fix: update 61 outdated test mocks to match current schemas

2026-03-24 06:40:42 +01:00

test_compliance_scope_routes.py

feat: Vorbereitung-Module auf 100% — Compliance-Scope Backend, DELETE-Endpoints, Proxy-Fixes, blocked-content Tab

2026-03-04 17:43:29 +01:00

test_consent_template_routes.py

feat: Betrieb-Module → 100% — Echte CRUD-Flows, kein Mock-Data

2026-03-03 12:48:43 +01:00

test_control_composer.py

feat(multi-layer): complete Multi-Layer Control Architecture (Phases 1-8 + Pass 0)

2026-03-17 09:00:37 +01:00

test_control_dedup.py

feat(rag): optimize RAG pipeline — JSON-Mode, CoT, Hybrid Search, Re-Ranking, Cross-Reg Dedup, chunk 1024

2026-03-21 11:49:43 +01:00

test_control_generator.py

feat(rag): optimize RAG pipeline — JSON-Mode, CoT, Hybrid Search, Re-Ranking, Cross-Reg Dedup, chunk 1024

2026-03-21 11:49:43 +01:00

test_control_patterns.py

feat(multi-layer): complete Multi-Layer Control Architecture (Phases 1-8 + Pass 0)

2026-03-17 09:00:37 +01:00

test_control_routes.py

feat: Anti-Fake-Evidence System (Phase 1-4b)

2026-03-23 17:15:45 +01:00

test_crosswalk_routes.py

feat(multi-layer): complete Multi-Layer Control Architecture (Phases 1-8 + Pass 0)

2026-03-17 09:00:37 +01:00

test_dashboard_routes_extended.py

feat: add compliance modules 2-5 (dashboard, security templates, process manager, evidence collector)

2026-03-14 21:03:04 +01:00

test_decomposition_pass.py

feat: Framework Decomposition Engine + Composite Detection for Pass 0b

2026-03-23 12:11:55 +01:00

test_document_versions.py

feat(sdk): Multi-Tenancy, Versionierung, Change-Requests, Dokumentengenerierung (Phase 1-6)

2026-03-07 14:12:34 +01:00

test_dsfa_routes.py

fix: update 61 outdated test mocks to match current schemas

2026-03-24 06:40:42 +01:00

test_dsr_routes.py

feat(dsr): Go DSR deprecated, Python Export-Endpoint, Frontend an Backend-APIs anbinden

2026-03-06 18:21:43 +01:00

test_einwilligungen_routes.py

feat: Rechtliche-Texte-Module auf 100% — Dead Code, RAG-Fallback, Fehler-UI

2026-03-03 22:27:13 +01:00

test_email_template_routes.py

feat: Consent-Service Module nach Compliance migriert (DSR, E-Mail-Templates, Legal Docs, Banner)

2026-03-05 00:36:24 +01:00

test_escalation_routes.py

feat(sdk): API-Referenz Frontend + Backend-Konsolidierung (Shared Utilities, CRUD Factory)

2026-03-07 17:07:43 +01:00

test_evidence_check_routes.py

feat: add compliance modules 2-5 (dashboard, security templates, process manager, evidence collector)

2026-03-14 21:03:04 +01:00

test_evidence_routes.py

feat: Anti-Fake-Evidence System (Phase 1-4b)

2026-03-23 17:15:45 +01:00

test_extraction_routes.py

feat(extraction): POST /compliance/extract-requirements-from-rag

2026-03-05 15:11:10 +01:00

test_framework_decomposition.py

feat: Framework Decomposition Engine + Composite Detection for Pass 0b

2026-03-23 12:11:55 +01:00

test_generation_routes.py

fix(quality): Ruff/CVE/TS-Fixes, 104 neue Tests, Complexity-Refactoring

2026-03-07 19:00:33 +01:00

test_import_routes.py

fix(import+screening): GET-Alias, DELETE-Endpoint, ehrlicher Scan-Status

2026-03-05 12:07:01 +01:00

test_incident_routes.py

feat(incidents): Go Incidents nach Python migrieren, Proxy umleiten, 50 Tests

2026-03-06 20:50:00 +01:00

test_isms_routes.py

fix(mock-data): Fake-Daten bei leerer DB entfernt — ISMS 0%, Dashboard keine simulierten Trends, Compliance-Hub keine Fallback-Zahlen

2026-03-07 23:25:25 +01:00

test_legal_document_routes_extended.py

fix: update 61 outdated test mocks to match current schemas

2026-03-24 06:40:42 +01:00

test_legal_document_routes.py

test: Regressionstests für Package 4 Phase 3 — ip_address/user_agent + Versions-Array-Format

2026-03-03 11:04:02 +01:00

test_legal_template_routes.py

fix: update 61 outdated test mocks to match current schemas

2026-03-24 06:40:42 +01:00

test_license_gate.py

feat(canonical-controls): Canonical Control Library — rechtssichere Security Controls

2026-03-12 19:55:06 +01:00

test_loeschfristen_routes.py

feat(sdk): vendor-compliance cross-module integration — VVT, obligations, TOM, loeschfristen

2026-03-19 13:59:43 +01:00

test_migration_060.py

feat(multi-layer): complete Multi-Layer Control Architecture (Phases 1-8 + Pass 0)

2026-03-17 09:00:37 +01:00

test_module_routes.py

feat(freigabe): Import/Screening/Modules/RAG — API-Tests, Migration 031, Bug-Fix

2026-03-05 11:42:19 +01:00

test_notfallplan_routes.py

feat: Alle 5 verbleibenden SDK-Module auf 100% — RAG, Security-Backlog, Quality, Notfallplan, Loeschfristen

2026-03-03 18:04:53 +01:00

test_obligation_extractor.py

feat(rag): optimize RAG pipeline — JSON-Mode, CoT, Hybrid Search, Re-Ranking, Cross-Reg Dedup, chunk 1024

2026-03-21 11:49:43 +01:00

test_obligation_routes.py

feat(sdk): vendor-compliance cross-module integration — VVT, obligations, TOM, loeschfristen

2026-03-19 13:59:43 +01:00

test_pattern_matcher.py

feat(multi-layer): complete Multi-Layer Control Architecture (Phases 1-8 + Pass 0)

2026-03-17 09:00:37 +01:00

test_pipeline_adapter.py

feat(multi-layer): complete Multi-Layer Control Architecture (Phases 1-8 + Pass 0)

2026-03-17 09:00:37 +01:00

test_policy_templates.py

fix: update 61 outdated test mocks to match current schemas

2026-03-24 06:40:42 +01:00

test_process_task_routes.py

feat: add compliance modules 2-5 (dashboard, security templates, process manager, evidence collector)

2026-03-14 21:03:04 +01:00

test_project_routes.py

feat(sdk): Multi-Projekt-Architektur — mehrere Projekte pro Tenant

2026-03-09 14:53:50 +01:00

test_quality_routes.py

feat(sdk): API-Referenz Frontend + Backend-Konsolidierung (Shared Utilities, CRUD Factory)

2026-03-07 17:07:43 +01:00

test_rag_client.py

…

test_requirement_routes.py

feat(compliance-kern): Tests, MkDocs + RAG-Controls Button für Anforderungen/Controls/Nachweise/Risiken

2026-03-05 13:43:02 +01:00

test_reranker.py

feat(rag): optimize RAG pipeline — JSON-Mode, CoT, Hybrid Search, Re-Ranking, Cross-Reg Dedup, chunk 1024

2026-03-21 11:49:43 +01:00

test_risk_routes.py

feat(compliance-kern): Tests, MkDocs + RAG-Controls Button für Anforderungen/Controls/Nachweise/Risiken

2026-03-05 13:43:02 +01:00

test_routes_legal_context.py

…

test_screening_routes.py

feat(freigabe): Import/Screening/Modules/RAG — API-Tests, Migration 031, Bug-Fix

2026-03-05 11:42:19 +01:00

test_security_backlog_routes.py

feat(sdk): API-Referenz Frontend + Backend-Konsolidierung (Shared Utilities, CRUD Factory)

2026-03-07 17:07:43 +01:00

test_security_templates.py

feat: add compliance modules 2-5 (dashboard, security templates, process manager, evidence collector)

2026-03-14 21:03:04 +01:00

test_source_policy_routes.py

docs: MkDocs, SDK-Flow und Tests fuer 6 Dokumentations-Module aktualisieren

2026-03-02 18:48:55 +01:00

test_tom_mapping_routes.py

feat(tom): audit document, compliance checks, 25 controls, canonical control mapping

2026-03-19 11:56:53 +01:00

test_tom_routes.py

feat(tom): TOM-Backend in Python erstellen, Frontend von In-Memory auf DB migrieren

2026-03-06 17:35:44 +01:00

test_validate_controls.py

feat(canonical-controls): Canonical Control Library — rechtssichere Security Controls

2026-03-12 19:55:06 +01:00

test_vendor_compliance_routes.py

docs: Qdrant und MinIO/Object-Storage Referenzen aktualisieren

2026-03-06 20:18:29 +01:00

test_vvt_library_routes.py

feat(sdk): VVT master libraries, process templates, Loeschfristen profiling + document

2026-03-19 11:56:25 +01:00

test_vvt_routes.py

fix(quality): Ruff/CVE/TS-Fixes, 104 neue Tests, Complexity-Refactoring

2026-03-07 19:00:33 +01:00

test_vvt_tenant_isolation.py

fix: update 61 outdated test mocks to match current schemas

2026-03-24 06:40:42 +01:00