Previous version searched against atomic_controls_dedup collection which
only contains Pass 0b atomic controls. Now creates a temporary collection
with ALL draft controls as reference, then checks targets against it.
Two phases:
1. Index ~53k reference drafts into temp Qdrant collection (batch 32)
2. Search each of 14k target controls, Embedding + LLM for borderline
3. Cleanup temp collection when done
Status updates every 50 controls (fixed counter bug).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
POST /generate/harmonization-recheck verifies promoted controls
against Qdrant dedup collection via Embedding + LLM. Runs as stable
asyncio background task inside the container (no docker exec issues).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds confidence scoring, escalation detection, and reasoning to the
deterministic filter. All assessment is deterministic (no LLM).
Confidence scoring (0.0-1.0):
- +0.25 industry specified
- +0.15 company size specified
- +0.20-0.30 scope signals provided
- +0.15 controls found
- +0.15 no contradictions
- Capped at 0.75 for escalation cases
Escalation triggers:
- Contradictory signals (holds_client_funds without operates_payment_service)
- Ambiguous signals (provides_embedded_connectivity)
- Financial signals without explicit payment service declaration
- Incomplete profile (no industry, size, or signals)
Reasoning: template-based, includes active signals, control count,
scope-condition descriptions, and warnings.
Response now includes "assessment" field with confidence, escalation_flag,
escalation_reason, inferred_signals, reasoning, and warnings.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Deleted 3 packages that were copied without validation:
- applicability_demo/ (fictional control IDs, wrong API schema)
- applicability_demo_sdk/ (wrong endpoint URL, fictional request format)
- applicability_demo_ci/ (GitHub Actions instead of Gitea, duplicated code)
Replaced with real integration in test_applicability_use_cases.py:
- TestApplicabilityIntegration calls real get_applicable_controls()
- Checks source_citation->source and control_id domain prefixes
- Runs against actual DB when DATABASE_URL is set
- 128 structure/acceptance tests pass, 24 integration tests skip without DB
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Makefile + pytest + GitHub Actions workflow for automated regression:
- make install / make eval / make test
- pytest integration with demo_cases.yaml
- Golden outputs for 6 priority cases
- Report generation (JSON + Markdown)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Request payloads + response contract + api_runner.py for 6 priority cases.
Can be run directly against /v1/applicability/evaluate endpoint.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Stage 4 (Harmonization) now uses two-tier approach:
- Score >= 0.92: auto-duplicate (embedding only, fast)
- Score 0.85-0.92: LLM verification via local qwen3.5 (think=false, ~3s)
- Score < 0.85: not a duplicate
This eliminates ~44% false positives from pure embedding similarity.
LLM_DEDUP_ENABLED env var controls the feature (default: true).
Also adds 10 applicability use case tests (bank+TAN, webshop+Stripe,
SaaS startup, energy provider, health app, automotive, law firm, etc.)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
POST /v1/canonical/generate/backfill-applicability enriches controls
with applicable_industries, applicable_company_size, scope_conditions
via Anthropic API. Targets ~26k controls from pipeline version < 3.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Ollama API supports "think": false to disable extended thinking mode
on qwen3.5. Reduces response time from 95s to ~3s per comparison.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
qwen3.5 uses extended thinking by default which causes 95s+ responses
and 30s timeouts. Add /no_think to system prompt and num_predict=200
to keep responses short.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
POST /v1/canonical/generate/llm-dedup uses local Ollama (qwen3.5:35b-a3b)
to verify borderline duplicate matches (score 0.85-0.91). More accurate
than embedding similarity for compliance controls with subtle scope
differences (e.g. "documented" vs "implemented").
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Stale UUIDs in the Qdrant dedup collection can reference controls
that were deprecated in earlier batches. Log warning and continue
instead of raising and killing the entire job.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add migration script (scripts/migrate_jsonb.py) that converts
89,443 Python dict repr rows to valid JSON via ast.literal_eval
- Column altered from TEXT to native JSONB
- Index created on generation_metadata->>'merge_group_hint'
- Remove unnecessary ::jsonb casts in pipeline_adapter.py
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
POST /v1/canonical/generate/backfill-repair uses Anthropic API to
generate missing fields from available context (source text, other
fields). Handles 1,470 controls with incomplete data.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Qdrant collections use regulation_id (not regulation_code), regulation_name_de,
guideline_name, download_url etc. Also search bp_compliance_datenschutz
collection where OWASP/ENISA docs live.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Go SDK RAG proxy returns 401 (Qdrant API key mismatch). Switch
AnchorFinder to use direct Qdrant vector search + embedding service,
same approach as the main pipeline. No dependency on Go SDK anymore.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add POST /v1/canonical/generate/backfill-anchors endpoint for batch
populating open_anchors on controls generated with skip_web_search=true
- Uses AnchorFinder Stage A (RAG search) to find OWASP/NIST/ENISA refs
- Background job with progress tracking (same pattern as other backfills)
- Promotes needs_review controls that gain anchors to draft state
- Target audience normalization (enterprise/authority/provider → JSON arrays)
already applied via SQL
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Batch-Postprocessing: Controls mit title/objective = None/null/"" werden
gefiltert und nicht gespeichert. Title wird aus Objective abgeleitet falls
nur Title fehlt.
- _store_control: Pre-store Quality Guard lehnt leere Controls ab
- Verhindert "None"-Controls die durch LLM-Parsing-Fehler entstehen
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root Cause: _generate_control_id erzeugte ID-Kollisionen (String-Sort statt
numeric), ON CONFLICT DO NOTHING verwarf Controls stillschweigend, Chunks
wurden als "processed" markiert obwohl Store fehlschlug → permanent verloren.
Fixes:
1. _generate_control_id: Numeric MAX statt String-Sort, Collision Guard
mit UUID-Suffix Fallback, Exception wird geloggt statt verschluckt
2. _store_control: ON CONFLICT DO UPDATE statt DO NOTHING → ID immer returned
3. Store-Logik: Chunk wird bei store_failed NICHT mehr als processed markiert
→ Retry beim naechsten Lauf moeglich
4. Counter: controls_generated nur bei erfolgreichem Store inkrementiert
Neue Counter: controls_stored + controls_store_failed
5. Anthropic API: HTTP 429/500/502/503/504 werden jetzt retried (2 Versuche)
6. Monitoring: Progress-Log zeigt Store-Rate (%), ALARM bei <80%
7. Post-Job Validierung: Vergleicht Generated vs Stored vs DB-Realitaet
WARNUNG wenn store_failed > 0, KRITISCH wenn Rate < 90%
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Anchor-Search (DuckDuckGo + RAG via SDK) verlangsamt Pipeline von
~50 Chunks/min auf ~10 Chunks/min. Anchors (OWASP/NIST-Referenzen)
koennen nachtraeglich in einem Batch-Job befuellt werden.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Control-Pipeline (Pass 0a/0b, BatchDedup, Generator) als eigenstaendiger
Service in Core, damit Compliance-Repo unabhaengig refakturiert werden kann.
Schreibt weiterhin ins compliance-Schema der shared PostgreSQL.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
