Allows switching between Haiku 4.5 and Sonnet 4.6 for Pass 0b
without rebuilding the backend container.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmark shows Haiku is 2.5x faster than Sonnet at 5x lower cost
for this JSON structuring task. Quality is equivalent.
$142 vs $705 for 75K obligations, ~2.8 days vs ~7 days.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous formula (batch_size * 1500) exceeded Claude's 16K output limit
for batch_size > 10, causing API failures and Ollama fallback.
New formula: min(16384, max(4096, batch_size * 500))
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- _write_atomic_control() now uses RETURNING id and inserts into
control_parent_links (M:N) with source_regulation, source_article,
and obligation_candidate_id parsed from parent's source_citation
- New _parse_citation() helper for JSONB source_citation extraction
- New GET /controls/{id}/traceability endpoint returning full chain:
parent links with obligations, child controls, source_count
- Backend: control_type filter (atomic/rich) for controls + count
- Frontend: Rechtsgrundlagen section in ControlDetail showing all
parent links per source regulation with obligation text + strength
- Frontend: Atomic/Rich filter dropdown in Control Library list
- Frontend: GenerationStrategyBadge recognizes 'pass0b' strategy
- Tests: 3 new tests for parent_link creation + citation parsing,
existing batch test mock updated for RETURNING clause
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TestAllowedCollections was asserting bp_compliance_recht which was
removed from the handler whitelist. Updated test to match the actual
AllowedCollections map (added bp_compliance_gdpr, bp_dsfa_templates,
bp_dsfa_risks, bp_iace_libraries; removed bp_compliance_recht).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Covers NIST SP 800-160/30/82, SPDX 3.0, CVSS v4.0, SLSA v1.0,
CycloneDX 1.6, OpenTelemetry, EU Machinery Guide 2006/42/EC,
FDA Human Factors, and 5 GPAI documents (Scope Guidelines,
Communication, CoP Safety/Transparency/Copyright).
All documents include license metadata in regulation payloads.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add obligation refinement pipeline between Pass 0a and 0b:
- Merge pass: rule-based dedup of implementation-level duplicate obligations
within the same parent control (Jaccard similarity on action+object)
- Enrich pass: classify trigger_type (event/periodic/continuous) and detect
is_implementation_specific from obligation text (regex-based, no LLM)
- Pass 0b: skip merged obligations, cap severity for impl-specific, override
category to 'testing' for test obligations
- Migration 075: merged_into_id, trigger_type, is_implementation_specific
- Two new API endpoints: merge-obligations, enrich-obligations
- 30+ new tests (122 total, all passing)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move torch and sentence-transformers to requirements-reranker.txt so
the main Docker build succeeds even if these large packages fail to
install. The reranker code already handles missing imports gracefully
when RERANK_ENABLED=false (the default).
This fixes the production deployment — builds were failing because of
the ~800MB torch dependency, preventing ALL new code from deploying.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Crosswalk routes returning 404 on production. This adds a diagnostic
endpoint that reports which sub-routers failed to load and why.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 1 (LLM Quality):
- Add format=json to all Ollama payloads (obligation_extractor, control_generator, citation_backfill)
- Add Chain-of-Thought analysis steps to Pass 0a/0b system prompts
Phase 2 (Retrieval Quality):
- Hybrid search via Qdrant Query API with RRF fusion + automatic text index (legal_rag.go)
- Fallback to dense-only search if Query API unavailable
- Cross-encoder re-ranking with BGE Reranker v2 (RERANK_ENABLED=false by default)
- CPU-only PyTorch dependency to keep Docker image small
Phase 3 (Data Layer):
- Cross-regulation dedup pass (threshold 0.95) links controls across regulations
- DedupResult.link_type field distinguishes dedup_merge vs cross_regulation
- Chunk size defaults updated 512/50 → 1024/128 for new ingestions only
- Existing collections and controls are NOT affected
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
v6.1.12 had an expired TrustedClientToken causing 403 on all Edge TTS
requests. v7.2.7 uses a valid token and same Communicate API.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Edge TTS provides near-human quality voices (de-DE-ConradNeural, en-US-GuyNeural).
Falls back to Piper TTS when Edge TTS is unavailable (e.g. no internet).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Download en_US-lessac-high Piper model in Dockerfile
- Select TTS engine based on request language (de/en)
- Include language in cache key to avoid collisions
- List both voices in /voices endpoint
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Returns MP3 audio directly in response body (no MinIO upload)
- Disk cache (/tmp/tts-cache) avoids re-synthesis of identical text
- Used by pitch-deck presenter for real-time TTS playback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- flow-data.ts: vendor-compliance moved from betrieb/seq:4200 to
dokumentation/seq:2500, prerequisite changed to vvt, added 5 DB tables
- architecture-data.ts: added vendor tables and API endpoints to
backend-compliance service definition
- StepHeader.tsx: added vendor-compliance explanation with 4 tips
(Art. 28, cross-module integration, third-country transfers, controls
library). Updated obligations (12 checks, vendor-link, document),
loeschfristen (vendor picker), tom (vendor-controls cross-ref),
vvt (processor tab from vendor API)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The HTML document builder was missing linked_vendor_ids in the detailed
obligation cards. Art. 28 obligations with linked vendors now display
them in the audit-ready PDF/HTML output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
QA process, article types, match rates, preamble dedup rules,
and next steps documented in MkDocs under Entwicklung.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Preamble controls that duplicate article controls (same regulation,
Jaccard title similarity >= 0.40) are marked as duplicate.
Article controls always take priority.
Result: 6,183 active controls (was 6,373), 648 unique preamble controls remain.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
QA pipeline that matches control source_original_text directly against
original PDF documents to verify article/paragraph assignments. Covers
backfill, dedup, source normalization, Qdrant cleanup, and prod sync.
Key results (2026-03-20):
- 4,110/7,943 controls matched to PDF (100% for major EU regs)
- 3,366 article corrections, 705 new assignments
- 1,290 controls from Erwägungsgründe (preamble) identified
- 779 controls from Anhänge (annexes) identified
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
eu_2023_1542 (Batterieverordnung), eu_2023_988 (GPSR), nist_sp800_218,
nist_privacy_1_0, owasp_mobile_top10 were defaulting to Rule 3 (restricted)
instead of their correct rules. This caused 68/71 controls to be flagged
as too_close in the last pipeline run.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add _detect_recital() to QA pipeline — flags controls where
source_original_text contains Erwägungsgrund markers instead of
article text (28% of controls with source text affected).
- Recital detection via regex + phrase matching in QA validation
- 10 new tests (TestRecitalDetection), 81 total
- ReviewCompare component for side-by-side duplicate comparison
- Review mode split: Duplikat-Verdacht vs Rule-3-ohne-Anchor tabs
- MkDocs: recital detection documentation
- Detection script for bulk analysis (scripts/find_recital_controls.py)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Updated both structure_batch and reformulate_batch prompts to return null
for chunks without actionable requirements (definitions, TOCs, scope-only).
Explicit instruction to always process annexes/appendices as they often
contain concrete technical requirements.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Local LLM prefilter (llama3.2 3B) was incorrectly skipping annex chunks
that contain concrete requirements. Added skip_prefilter flag to bypass
the local pre-filter and send all chunks directly to Anthropic API.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The embedding pre-load for 4998 existing controls takes ~16 minutes,
causing the SQLAlchemy session to become invalid. Added rollback after
pre-load completes to reset the session before subsequent DB operations.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Chunks without a regulation_code were silently passing through the filter
in _scan_rag(), causing unrelated documents (e.g. Data Act, legal templates)
to be included in filtered generation jobs. Now chunks without reg_code are
skipped when regulation_filter is active.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add Anthropic API support to decomposition Pass 0a/0b (prompt caching, content batching)
- Add Anthropic Batch API (50% cost reduction, async 24h processing)
- Add source_filter (ILIKE on source_citation) for regulation-based filtering
- Add category_filter to Pass 0a for selective decomposition
- Add regulation_filter to control_generator for RAG scan phase filtering
(prefix match on regulation_code — enables CE + Code Review focus)
- New API endpoints: batch-submit-0a, batch-submit-0b, batch-status, batch-process
- 83 new tests (all passing)
Cost reduction: $2,525 → ~$600-700 with all optimizations combined.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 5 — Frontend Integration:
- components/page.tsx: ComponentLibraryModal with 120 components + 20 energy sources
- hazards/page.tsx: AutoSuggestPanel with 3-column pattern matching review
- mitigations/page.tsx: SuggestMeasuresModal per hazard with 3-level grouping
- verification/page.tsx: SuggestEvidenceModal per mitigation with evidence types
Phase 6 — RAG Library Search:
- Added bp_iace_libraries to AllowedCollections whitelist in rag_handlers.go
- SearchLibrary endpoint: POST /iace/library-search (semantic search across libraries)
- EnrichTechFileSection endpoint: POST /projects/:id/tech-file/:section/enrich
- Created ingest-iace-libraries.sh ingestion script for Qdrant collection
Tests (123 passing):
- tag_taxonomy_test.go: 8 tests for taxonomy entries, domains, essential tags
- controls_library_test.go: 7 tests for measures, reduction types, subtypes
- integration_test.go: 7 integration tests for full match flow and library consistency
- Extended tag_resolver_test.go: 9 new tests for FindByTags and cross-category resolution
Documentation:
- Updated iace.md with Hazard-Matching-Engine, RAG enrichment, and new DB tables
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements Phases 1-4 of the IACE Hazard-Matching-Engine:
- 120 machine components (C001-C120) in 11 categories
- 20 energy sources (EN01-EN20)
- ~85 tag taxonomy across 5 domains
- 44 hazard patterns with AND/NOT matching logic
- Pattern engine with tag resolution and confidence scoring
- 8 new API endpoints (component-library, energy-sources, tags, patterns, match/apply)
- Completeness gate G09 for pattern matching
- 320 tests passing (36 new)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Group chunks by regulation_code before batching for better LLM context
- Add generation_strategy column (ungrouped=v1, document_grouped=v2)
- Add v1/v2 badge to control cards in frontend
- Add sort-by-source option with visual group headers
- Add frontend page tests (18 tests)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add new "Filter in der Control Library" section explaining all 7 dropdowns
- Add "Badges & Lizenzregeln" section explaining Rule 1/2/3 and all badges
- Update taxonomy with actual top-10 domains and counts (~3100+ controls)
- Update Master Library strategy with current numbers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
