- Add assertion, pass_criteria, fail_criteria, check_type to AtomicControlCandidate dataclass
- Parse MCP fields from LLM output in _process_pass0b_control
- Store MCP fields in generation_metadata JSON for later use by MCP scanner
- Fields default to empty when not present (backward-compatible with old prompts)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Obligations classified before API call:
- evidence → skipped (saves API cost)
- composite → skipped (not atomic)
- framework_container → skipped (decompose separately)
- atomic → sent to LLM
Filter stats returned in submit response.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fixes from v2 evaluation (7.9/10 avg, 28 controls):
1. COMPOUND BAN: "durchführen UND Maßnahmen ergreifen" → pick primary action only
2. EVIDENCE-OF-ACTION: "Tests dokumentieren" → evidence field, not own control
3. PFLICHT=PROZESS: "Behörden informieren" + "Verfahren etablieren" = 1 control
4. MERGE-KEY BUG: merge_key from LLM output now stored in generation_metadata
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Key changes to system prompt:
- Evidence/documentation belongs in evidence field, NOT as separate control
- SBOM = 1 control (not "maintain" + "document" separately)
- Security lifecycle phases (identify/assess/remediate/monitor) = separate controls
- Same object + same action + same actor = 1 control (merge, not split)
- Titles must contain the ACTION, not just the subject
WRONG: "Vertraulichkeit Mitarbeiter"
RIGHT: "Mitarbeiter zur Vertraulichkeit verpflichten"
Titles serve as MCP search queries against customer documents/code.
Bad titles = bad search results = unusable product.
All 52,566 old pass0b controls deprecated (not deleted) for full regeneration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Prevents UniqueViolation from blocking entire batch. Each result
is committed individually, errors are rolled back without affecting
subsequent results.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds confidence scoring, escalation detection, and reasoning to the
deterministic filter. All assessment is deterministic (no LLM).
Confidence scoring (0.0-1.0):
- +0.25 industry specified
- +0.15 company size specified
- +0.20-0.30 scope signals provided
- +0.15 controls found
- +0.15 no contradictions
- Capped at 0.75 for escalation cases
Escalation triggers:
- Contradictory signals (holds_client_funds without operates_payment_service)
- Ambiguous signals (provides_embedded_connectivity)
- Financial signals without explicit payment service declaration
- Incomplete profile (no industry, size, or signals)
Reasoning: template-based, includes active signals, control count,
scope-condition descriptions, and warnings.
Response now includes "assessment" field with confidence, escalation_flag,
escalation_reason, inferred_signals, reasoning, and warnings.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Stage 4 (Harmonization) now uses two-tier approach:
- Score >= 0.92: auto-duplicate (embedding only, fast)
- Score 0.85-0.92: LLM verification via local qwen3.5 (think=false, ~3s)
- Score < 0.85: not a duplicate
This eliminates ~44% false positives from pure embedding similarity.
LLM_DEDUP_ENABLED env var controls the feature (default: true).
Also adds 10 applicability use case tests (bank+TAN, webshop+Stripe,
SaaS startup, energy provider, health app, automotive, law firm, etc.)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Stale UUIDs in the Qdrant dedup collection can reference controls
that were deprecated in earlier batches. Log warning and continue
instead of raising and killing the entire job.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add migration script (scripts/migrate_jsonb.py) that converts
89,443 Python dict repr rows to valid JSON via ast.literal_eval
- Column altered from TEXT to native JSONB
- Index created on generation_metadata->>'merge_group_hint'
- Remove unnecessary ::jsonb casts in pipeline_adapter.py
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Qdrant collections use regulation_id (not regulation_code), regulation_name_de,
guideline_name, download_url etc. Also search bp_compliance_datenschutz
collection where OWASP/ENISA docs live.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Go SDK RAG proxy returns 401 (Qdrant API key mismatch). Switch
AnchorFinder to use direct Qdrant vector search + embedding service,
same approach as the main pipeline. No dependency on Go SDK anymore.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add POST /v1/canonical/generate/backfill-anchors endpoint for batch
populating open_anchors on controls generated with skip_web_search=true
- Uses AnchorFinder Stage A (RAG search) to find OWASP/NIST/ENISA refs
- Background job with progress tracking (same pattern as other backfills)
- Promotes needs_review controls that gain anchors to draft state
- Target audience normalization (enterprise/authority/provider → JSON arrays)
already applied via SQL
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Batch-Postprocessing: Controls mit title/objective = None/null/"" werden
gefiltert und nicht gespeichert. Title wird aus Objective abgeleitet falls
nur Title fehlt.
- _store_control: Pre-store Quality Guard lehnt leere Controls ab
- Verhindert "None"-Controls die durch LLM-Parsing-Fehler entstehen
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root Cause: _generate_control_id erzeugte ID-Kollisionen (String-Sort statt
numeric), ON CONFLICT DO NOTHING verwarf Controls stillschweigend, Chunks
wurden als "processed" markiert obwohl Store fehlschlug → permanent verloren.
Fixes:
1. _generate_control_id: Numeric MAX statt String-Sort, Collision Guard
mit UUID-Suffix Fallback, Exception wird geloggt statt verschluckt
2. _store_control: ON CONFLICT DO UPDATE statt DO NOTHING → ID immer returned
3. Store-Logik: Chunk wird bei store_failed NICHT mehr als processed markiert
→ Retry beim naechsten Lauf moeglich
4. Counter: controls_generated nur bei erfolgreichem Store inkrementiert
Neue Counter: controls_stored + controls_store_failed
5. Anthropic API: HTTP 429/500/502/503/504 werden jetzt retried (2 Versuche)
6. Monitoring: Progress-Log zeigt Store-Rate (%), ALARM bei <80%
7. Post-Job Validierung: Vergleicht Generated vs Stored vs DB-Realitaet
WARNUNG wenn store_failed > 0, KRITISCH wenn Rate < 90%
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Anchor-Search (DuckDuckGo + RAG via SDK) verlangsamt Pipeline von
~50 Chunks/min auf ~10 Chunks/min. Anchors (OWASP/NIST-Referenzen)
koennen nachtraeglich in einem Batch-Job befuellt werden.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Control-Pipeline (Pass 0a/0b, BatchDedup, Generator) als eigenstaendiger
Service in Core, damit Compliance-Repo unabhaengig refakturiert werden kann.
Schreibt weiterhin ins compliance-Schema der shared PostgreSQL.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
