The NOT EXISTS check and Duplicate Guard now exclude deprecated and
duplicate controls, enabling clean re-runs after invalidation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Truncate object keys to 40 chars (was 80) at underscore boundary
- Strip German qualifying prepositional phrases (bei/für/gemäß/von/zur/...)
- Add 65 new synonym mappings for near-duplicate patterns found in analysis
- Strip trailing noise tokens (articles/prepositions)
- Add _truncate_at_boundary() helper and _QUALIFYING_PHRASE_RE regex
- 11 new tests for normalization improvements (227 total pass)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. Duplicate Guard: merge_hint-Lookup vor INSERT in _write_atomic_control()
verhindert semantisch identische Controls unter demselben Parent.
2. Severity-Kalibrierung: action_type-basiert statt blind vom Parent.
define/review/test → max medium, implement/monitor → max high.
3. Title-Truncation: Schnitt am Wortende statt mitten im Wort.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Neue Endpunkte POST /obligations/dedup und GET /obligations/dedup-stats.
Pro candidate_id wird der aelteste Eintrag behalten, alle weiteren erhalten
release_state='duplicate' mit merged_into_id + quality_flags fuer Traceability.
Detail-View filtert Duplikate aus. MKDocs aktualisiert.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Backend: Facets zaehlen jetzt Controls OHNE Wert (z.B. "Ohne Nachweis")
als __none__. Filter unterstuetzen __none__ fuer verification_method,
category, evidence_type. Counts addieren sich immer zum Total.
Frontend: "Ohne X" Optionen in Dropdowns. AbortController verhindert
dass aeltere API-Antworten neuere ueberschreiben.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Backend: controls-meta akzeptiert alle Filter-Parameter und berechnet
Faceted Counts (jede Dimension zaehlt mit allen ANDEREN Filtern).
Neue Facets: severity, verification_method, category, evidence_type,
release_state — zusaetzlich zu domains, sources, type_counts.
Frontend: loadMeta laedt bei jeder Filteraenderung neu, alle Dropdowns
zeigen kontextsensitive Zahlen. Proxy leitet Filter an controls-meta weiter.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Backend: control_type=eigenentwicklung in list_controls + count_controls,
type_counts (rich/atomic/eigenentwicklung) in controls-meta Endpoint.
Frontend: Typ-Dropdown zeigt Eigenentwicklung mit Anzahl.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Die atomic_controls_dedup Collection (51k Punkte) enthaelt nur atomare
Controls ohne source_citation. Jetzt wird der Parent-Control aufgeloest,
der die Rechtsgrundlage traegt. Deduplizierung nach Parent-UUID verhindert
mehrfache Eintraege fuer die gleiche Regulation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Verhindert 'invalid transaction' Fehler wenn ein LLM-Call fehlschlaegt
und nachfolgende DB-Operationen blockiert.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
qwen3.5 gibt Antworten im 'thinking'-Feld statt 'response' zurueck.
Mit think:false wird der Thinking-Mode deaktiviert und die Antwort
korrekt im response-Feld geliefert.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ollama als eigener Enum-Wert neben self_hosted, damit die
docker-compose-Konfiguration (ollama) korrekt aufgeloest wird.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
POST /controls/backfill-rationale — ersetzt Placeholder "Aus Obligation
abgeleitet." durch LLM-generierte Begruendungen (Ollama/qwen3.5).
Optimierung: gruppiert ~86k Controls nach ~7k Parents, ein LLM-Call pro Parent.
Paginierung via batch_size/offset fuer kontrollierte Ausfuehrung.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DB-Constraint erlaubt nur must/should/may. 'can' gibt es nicht.
Alle Referenzen auf 'can' durch 'may' ersetzt.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Die Obligation kennt ihren Parent-Rich-Control direkt. Dessen
source_citation->>'source' gibt die Quell-Regulierung zuverlaessiger
als der Umweg ueber control_parent_links (M:N-Inflation).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Backend: provenance endpoint (obligations, doc refs, merged duplicates,
regulations summary) + atomic-stats aggregation endpoint.
Frontend: ControlDetail mit Provenance-Sektionen, klickbare Navigation,
neue /sdk/atomic-controls Seite mit Stats-Bar und gefilterer Liste.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SQLAlchemy's text() parser doesn't properly handle :param::type
syntax — it fails to recognize :dd as a bind parameter when followed
by ::jsonb. Using CAST(:dd AS jsonb) instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SQLAlchemy sessions enter a failed state after SQL errors.
Without rollback(), all subsequent queries on the same session
fail with InFailedSqlTransaction. Added try/except with rollback
in _mark_duplicate, _mark_duplicate_to, _write_review, cross-group
pass, and the main phase1 loop.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All Pass 0b controls have pattern_id=NULL. Rewritten to:
- Phase 1: Group by merge_group_hint (action:object:trigger), 52k groups
- Phase 2: Cross-group embedding search for semantically similar masters
- Qdrant search uses unfiltered cross-regulation endpoint
- API param changed: pattern_id → hint_filter
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmark shows Haiku is 2.5x faster than Sonnet at 5x lower cost
for this JSON structuring task. Quality is equivalent.
$142 vs $705 for 75K obligations, ~2.8 days vs ~7 days.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous formula (batch_size * 1500) exceeded Claude's 16K output limit
for batch_size > 10, causing API failures and Ollama fallback.
New formula: min(16384, max(4096, batch_size * 500))
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- _write_atomic_control() now uses RETURNING id and inserts into
control_parent_links (M:N) with source_regulation, source_article,
and obligation_candidate_id parsed from parent's source_citation
- New _parse_citation() helper for JSONB source_citation extraction
- New GET /controls/{id}/traceability endpoint returning full chain:
parent links with obligations, child controls, source_count
- Backend: control_type filter (atomic/rich) for controls + count
- Frontend: Rechtsgrundlagen section in ControlDetail showing all
parent links per source regulation with obligation text + strength
- Frontend: Atomic/Rich filter dropdown in Control Library list
- Frontend: GenerationStrategyBadge recognizes 'pass0b' strategy
- Tests: 3 new tests for parent_link creation + citation parsing,
existing batch test mock updated for RETURNING clause
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add obligation refinement pipeline between Pass 0a and 0b:
- Merge pass: rule-based dedup of implementation-level duplicate obligations
within the same parent control (Jaccard similarity on action+object)
- Enrich pass: classify trigger_type (event/periodic/continuous) and detect
is_implementation_specific from obligation text (regex-based, no LLM)
- Pass 0b: skip merged obligations, cap severity for impl-specific, override
category to 'testing' for test obligations
- Migration 075: merged_into_id, trigger_type, is_implementation_specific
- Two new API endpoints: merge-obligations, enrich-obligations
- 30+ new tests (122 total, all passing)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Crosswalk routes returning 404 on production. This adds a diagnostic
endpoint that reports which sub-routers failed to load and why.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 1 (LLM Quality):
- Add format=json to all Ollama payloads (obligation_extractor, control_generator, citation_backfill)
- Add Chain-of-Thought analysis steps to Pass 0a/0b system prompts
Phase 2 (Retrieval Quality):
- Hybrid search via Qdrant Query API with RRF fusion + automatic text index (legal_rag.go)
- Fallback to dense-only search if Query API unavailable
- Cross-encoder re-ranking with BGE Reranker v2 (RERANK_ENABLED=false by default)
- CPU-only PyTorch dependency to keep Docker image small
Phase 3 (Data Layer):
- Cross-regulation dedup pass (threshold 0.95) links controls across regulations
- DedupResult.link_type field distinguishes dedup_merge vs cross_regulation
- Chunk size defaults updated 512/50 → 1024/128 for new ingestions only
- Existing collections and controls are NOT affected
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
eu_2023_1542 (Batterieverordnung), eu_2023_988 (GPSR), nist_sp800_218,
nist_privacy_1_0, owasp_mobile_top10 were defaulting to Rule 3 (restricted)
instead of their correct rules. This caused 68/71 controls to be flagged
as too_close in the last pipeline run.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add _detect_recital() to QA pipeline — flags controls where
source_original_text contains Erwägungsgrund markers instead of
article text (28% of controls with source text affected).
- Recital detection via regex + phrase matching in QA validation
- 10 new tests (TestRecitalDetection), 81 total
- ReviewCompare component for side-by-side duplicate comparison
- Review mode split: Duplikat-Verdacht vs Rule-3-ohne-Anchor tabs
- MkDocs: recital detection documentation
- Detection script for bulk analysis (scripts/find_recital_controls.py)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Updated both structure_batch and reformulate_batch prompts to return null
for chunks without actionable requirements (definitions, TOCs, scope-only).
Explicit instruction to always process annexes/appendices as they often
contain concrete technical requirements.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Local LLM prefilter (llama3.2 3B) was incorrectly skipping annex chunks
that contain concrete requirements. Added skip_prefilter flag to bypass
the local pre-filter and send all chunks directly to Anthropic API.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The embedding pre-load for 4998 existing controls takes ~16 minutes,
causing the SQLAlchemy session to become invalid. Added rollback after
pre-load completes to reset the session before subsequent DB operations.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Chunks without a regulation_code were silently passing through the filter
in _scan_rag(), causing unrelated documents (e.g. Data Act, legal templates)
to be included in filtered generation jobs. Now chunks without reg_code are
skipped when regulation_filter is active.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add Anthropic API support to decomposition Pass 0a/0b (prompt caching, content batching)
- Add Anthropic Batch API (50% cost reduction, async 24h processing)
- Add source_filter (ILIKE on source_citation) for regulation-based filtering
- Add category_filter to Pass 0a for selective decomposition
- Add regulation_filter to control_generator for RAG scan phase filtering
(prefix match on regulation_code — enables CE + Code Review focus)
- New API endpoints: batch-submit-0a, batch-submit-0b, batch-status, batch-process
- 83 new tests (all passing)
Cost reduction: $2,525 → ~$600-700 with all optimizations combined.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements Phases 1-4 of the IACE Hazard-Matching-Engine:
- 120 machine components (C001-C120) in 11 categories
- 20 energy sources (EN01-EN20)
- ~85 tag taxonomy across 5 domains
- 44 hazard patterns with AND/NOT matching logic
- Pattern engine with tag resolution and confidence scoring
- 8 new API endpoints (component-library, energy-sources, tags, patterns, match/apply)
- Completeness gate G09 for pattern matching
- 320 tests passing (36 new)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
