Benchmark shows Haiku is 2.5x faster than Sonnet at 5x lower cost
for this JSON structuring task. Quality is equivalent.
$142 vs $705 for 75K obligations, ~2.8 days vs ~7 days.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous formula (batch_size * 1500) exceeded Claude's 16K output limit
for batch_size > 10, causing API failures and Ollama fallback.
New formula: min(16384, max(4096, batch_size * 500))
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- _write_atomic_control() now uses RETURNING id and inserts into
control_parent_links (M:N) with source_regulation, source_article,
and obligation_candidate_id parsed from parent's source_citation
- New _parse_citation() helper for JSONB source_citation extraction
- New GET /controls/{id}/traceability endpoint returning full chain:
parent links with obligations, child controls, source_count
- Backend: control_type filter (atomic/rich) for controls + count
- Frontend: Rechtsgrundlagen section in ControlDetail showing all
parent links per source regulation with obligation text + strength
- Frontend: Atomic/Rich filter dropdown in Control Library list
- Frontend: GenerationStrategyBadge recognizes 'pass0b' strategy
- Tests: 3 new tests for parent_link creation + citation parsing,
existing batch test mock updated for RETURNING clause
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add obligation refinement pipeline between Pass 0a and 0b:
- Merge pass: rule-based dedup of implementation-level duplicate obligations
within the same parent control (Jaccard similarity on action+object)
- Enrich pass: classify trigger_type (event/periodic/continuous) and detect
is_implementation_specific from obligation text (regex-based, no LLM)
- Pass 0b: skip merged obligations, cap severity for impl-specific, override
category to 'testing' for test obligations
- Migration 075: merged_into_id, trigger_type, is_implementation_specific
- Two new API endpoints: merge-obligations, enrich-obligations
- 30+ new tests (122 total, all passing)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move torch and sentence-transformers to requirements-reranker.txt so
the main Docker build succeeds even if these large packages fail to
install. The reranker code already handles missing imports gracefully
when RERANK_ENABLED=false (the default).
This fixes the production deployment — builds were failing because of
the ~800MB torch dependency, preventing ALL new code from deploying.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Crosswalk routes returning 404 on production. This adds a diagnostic
endpoint that reports which sub-routers failed to load and why.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 1 (LLM Quality):
- Add format=json to all Ollama payloads (obligation_extractor, control_generator, citation_backfill)
- Add Chain-of-Thought analysis steps to Pass 0a/0b system prompts
Phase 2 (Retrieval Quality):
- Hybrid search via Qdrant Query API with RRF fusion + automatic text index (legal_rag.go)
- Fallback to dense-only search if Query API unavailable
- Cross-encoder re-ranking with BGE Reranker v2 (RERANK_ENABLED=false by default)
- CPU-only PyTorch dependency to keep Docker image small
Phase 3 (Data Layer):
- Cross-regulation dedup pass (threshold 0.95) links controls across regulations
- DedupResult.link_type field distinguishes dedup_merge vs cross_regulation
- Chunk size defaults updated 512/50 → 1024/128 for new ingestions only
- Existing collections and controls are NOT affected
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
eu_2023_1542 (Batterieverordnung), eu_2023_988 (GPSR), nist_sp800_218,
nist_privacy_1_0, owasp_mobile_top10 were defaulting to Rule 3 (restricted)
instead of their correct rules. This caused 68/71 controls to be flagged
as too_close in the last pipeline run.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add _detect_recital() to QA pipeline — flags controls where
source_original_text contains Erwägungsgrund markers instead of
article text (28% of controls with source text affected).
- Recital detection via regex + phrase matching in QA validation
- 10 new tests (TestRecitalDetection), 81 total
- ReviewCompare component for side-by-side duplicate comparison
- Review mode split: Duplikat-Verdacht vs Rule-3-ohne-Anchor tabs
- MkDocs: recital detection documentation
- Detection script for bulk analysis (scripts/find_recital_controls.py)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Updated both structure_batch and reformulate_batch prompts to return null
for chunks without actionable requirements (definitions, TOCs, scope-only).
Explicit instruction to always process annexes/appendices as they often
contain concrete technical requirements.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Local LLM prefilter (llama3.2 3B) was incorrectly skipping annex chunks
that contain concrete requirements. Added skip_prefilter flag to bypass
the local pre-filter and send all chunks directly to Anthropic API.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The embedding pre-load for 4998 existing controls takes ~16 minutes,
causing the SQLAlchemy session to become invalid. Added rollback after
pre-load completes to reset the session before subsequent DB operations.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Chunks without a regulation_code were silently passing through the filter
in _scan_rag(), causing unrelated documents (e.g. Data Act, legal templates)
to be included in filtered generation jobs. Now chunks without reg_code are
skipped when regulation_filter is active.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add Anthropic API support to decomposition Pass 0a/0b (prompt caching, content batching)
- Add Anthropic Batch API (50% cost reduction, async 24h processing)
- Add source_filter (ILIKE on source_citation) for regulation-based filtering
- Add category_filter to Pass 0a for selective decomposition
- Add regulation_filter to control_generator for RAG scan phase filtering
(prefix match on regulation_code — enables CE + Code Review focus)
- New API endpoints: batch-submit-0a, batch-submit-0b, batch-status, batch-process
- 83 new tests (all passing)
Cost reduction: $2,525 → ~$600-700 with all optimizations combined.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements Phases 1-4 of the IACE Hazard-Matching-Engine:
- 120 machine components (C001-C120) in 11 categories
- 20 energy sources (EN01-EN20)
- ~85 tag taxonomy across 5 domains
- 44 hazard patterns with AND/NOT matching logic
- Pattern engine with tag resolution and confidence scoring
- 8 new API endpoints (component-library, energy-sources, tags, patterns, match/apply)
- Completeness gate G09 for pattern matching
- 320 tests passing (36 new)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Group chunks by regulation_code before batching for better LLM context
- Add generation_strategy column (ungrouped=v1, document_grouped=v2)
- Add v1/v2 badge to control cards in frontend
- Add sort-by-source option with visual group headers
- Add frontend page tests (18 tests)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Backfill 81 controls with empty source_citation.source from generation_metadata
- Add fallback to generation_metadata.source_regulation in ControlDetail blue box
- Improve Rule 3 amber box text for reformulated controls
- Add 30 new tests for batch processing (TestParseJsonArray, TestBatchSizeConfig,
TestBatchProcessingLoop) — all 61 control generator tests passing
- Fix stale test_config_defaults assertion (max_controls 50→0)
- Update canonical-control-library.md with batch processing pipeline docs,
processed chunks tracking, migration guide, and stats endpoint
- Update testing.md with canonical control generator test section
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New test_policy_templates.py: 67 tests covering all 29 policy types,
API creation, filtering, placeholders, seed script validation
- Updated test_legal_template_routes.py: fix type count 16→52
- New MKDocs page policy-bibliothek.md with full template reference
- Updated dokumentengenerierung.md and rechtliche-texte.md with cross-refs
- Added policy-bibliothek to mkdocs.yml navigation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 29 new document types (IT security, data, personnel, vendor, BCM
policies) to VALID_DOCUMENT_TYPES and 5 category pills to the document
generator UI. Include seed script for production DB population.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: migrations 046-047 used explicit BEGIN/COMMIT which
conflicts with psycopg2 implicit transactions, and ALTER TABLE
on canonical_controls fails when the table doesn't exist on
production. This blocked all subsequent migrations (048-053).
Changes:
- migration_runner.py: strip BEGIN/COMMIT from SQL before executing
- 046: wrap canonical_controls ALTER in DO $$ IF EXISTS block
- 047: wrap canonical_controls ALTER in DO $$ IF EXISTS block
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously, a single failed migration would abort all subsequent
migrations via raise RuntimeError. Now the runner logs the failure
and continues with remaining migrations, so independent schema
changes (e.g. 050-053) are not blocked by an unrelated failure
in an earlier migration (e.g. 048).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Migrations 048 and 049 reference canonical_processed_chunks and
canonical_controls tables which may not exist on all environments.
Wrap ALTER TABLE statements in DO blocks that check for table
existence first. This unblocks migrations 050-053 on production.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Many regulation codes (nist_sp800_53r5, eucsa, owasp_top10_2021, EDPB
guidelines, EU laws, AT/FR/ES/NL/IT/HU laws) were defaulting to Rule 3
(restricted) because they weren't in REGULATION_LICENSE_MAP. Now all
~100 regulation codes from RAG are properly mapped to Rule 1 or 2.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaced bare imports with safe_import_router pattern — if one sub-router
fails to import (e.g. missing dependency), other routers still load.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
from __future__ import annotations breaks Pydantic BaseModel runtime type
evaluation. Replaced str | None → Optional[str], list[str] → List[str] etc.
in control_generator.py, anchor_finder.py, control_generator_routes.py.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds migration_runner.py that executes pending migrations from
migrations/ directory when backend-compliance starts. Tracks applied
migrations in _migration_history table.
Handles existing databases: detects if tables from migrations 001-045
already exist and seeds the history table accordingly, so only new
migrations (046+) are applied.
Skippable via SKIP_MIGRATIONS=true env var.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Migration 045: Seed 10 controls (AUTH, NET, SUP, LOG, WEB, DATA, CRYP, REL)
with 39 open-source anchors into the database
- Backend: POST/PUT/DELETE endpoints for canonical controls CRUD
- Frontend proxy: PUT and DELETE methods added to canonical route
- Frontend: Control Library with create/edit/delete UI, full form with
open anchor management, scope, requirements, evidence, test procedures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Step 2 im VVT-Generator: Ja/Nein-Buttons durch expandierbare Kacheln ersetzt.
Pro Abteilung werden typische Datenkategorien als Checkboxen angezeigt (isTypical
vorausgefuellt), Art. 9 Kategorien orange hervorgehoben mit DSGVO-Warnung.
7 neue Wiki-Artikel fuer Datenkategorien pro Geschaeftsbereich (HR, Finanzen,
Vertrieb, Marketing, Support, IT, Produktion).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Problem: Company Profile nutzte hartcodiertes tenant_id=default ohne project_id.
Beim Wechsel zwischen Projekten wurden immer die gleichen (oder keine) Daten geladen.
Aenderungen:
- Migration 042: project_id Spalte + UNIQUE(tenant_id, project_id) Constraint,
fehlende Spalten (offering_urls, Adressfelder) nachgetragen
- Backend: Alle Queries nutzen WHERE tenant_id + project_id IS NOT DISTINCT FROM
- Proxy: project_id Query-Parameter wird durchgereicht
- Frontend: projectId aus SDK-Context, profileApiUrl() Helper fuer alle API-Aufrufe
- "Weiter" speichert jetzt immer den Draft (war schon so, ging aber ins Leere)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
