feat(compliance-check): MC-Classification + Embedding + Vendor-Redundanz + Action-Recipes + Borlabs-Features

Massiv-Update auf Basis BMW-Test-Iterationen (v1→v9):

Core Compliance-Check
- Sonnet check_type Klassifikation: text/process/review fuer alle 1874 MCs
  in compliance.doc_check_controls (script + Sidecar /data/mc_classification.db).
  rag_document_checker filtert auf check_type='text' fuer doc_check.
  Plus fits_doc_type-Audit (v2) + ui_only-Audit fuer DSA/E-Commerce-MCs in
  falscher doc_type-Schublade.
- scope_requires-Filter: biometric/ai_decision/child_targeting MCs werden
  per business_profile gefiltert (FRT skipped fuer BMW etc.).
- Embedding-Match (BGE-M3) als Phase-3 nach Regex-Match:
  Per-doc_type-Threshold-Override (impressum 0.50, dse/cookie 0.60),
  Short-Field-Rescue (15-Wort-Chunks) fuer Pflichtfelder im Impressum.
  Title+check_question als Embedding-Input fuer mehr Kontext.
- Cookie-Text-Routing: consent-tester gibt cmp_cookie_text aus dem
  CMP-Reconstruct zurueck, Backend bevorzugt das gegen DOM-Extraction
  wenn richer (BMW 1824 vs 600 Worte).

Vendor-Redundanz + EU-Alternativen + Cost-Saving
- vendor_redundancy.analyze() — funktionale Kategorisierung der CMP-Vendors,
  Detektion von Mehrfach-Anbietern pro Kategorie, EU-Alternative-Lookup
  (Matomo, IONOS, HERE, Friendly Captcha, Smart AdServer, ...).
- vendor_cost_estimator: Tier-Inferenz aus Cookie-Footprint (Cookie-Anzahl
  + Premium-Feature-Cookies + Third-Party-Quote → starter/professional/
  enterprise/premier).
- Self-Service-Werbung (Google/Meta/Pinterest/...) = 0 Lizenz-Kosten
  (nur Media-Spend, separat). DSP-Plattformen behalten enge Range.
- Tier-aware Saving-Range: bei Enterprise/Premier nutzen wir den
  oberen 40-100%-Band der Listpreise, nicht starter→premier.
- Multi-Function-Tools (Matomo Pro, SAP CX, IONOS Cloud, Userlike, Smart
  AdServer, HERE Maps, Vimeo Pro, LamaPoll) — ein Tool ersetzt mehrere
  Kategorien gleichzeitig.

Cookie-Wissens-DB + Funktionale Klassifikation
- cookie_knowledge_db: 50 kuratierte Top-Cookies (Google/Meta/Adobe/MS/...)
  mit vendor, exact_purpose, data_collected, IAB-TCF-IDs, reid_risk,
  schrems_ii_status, EuGH-Urteile, EU-Alternative.
- cookie_function_classifier: pro Cookie funktionale Rolle (tracking_id,
  ad_pixel, session_id, ab_test, csrf, ...) + blocking_impact.

Country-Inferenz aus Rechtsform
- cookie_link_validator: Country-Field wird aus Vendor-Name abgeleitet
  (A/S=DK, GmbH=DE, Inc=US, B.V.=NL, ...) plus Vendor-Lookup-Table.
  Reduziert false-positive no_country-Flags bei eindeutig-EU-Vendors
  (Adform DK, Pinterest IE).

Action-Recipes + Doc-Anchor-Locator
- finding_action_recipes: pro Finding-Typ (no_cookies_listed, no_country,
  broken_opt_out, "Auftragsverarbeiter erwaehnen", "Art. 22 Profiling",
  ...) eine strukturierte Anweisung mit what/why/fix_text/where/example.
  Zum 1:1-Einfuegen in Kunden-Dokumente.
- doc_anchor_locator: Embedding-basiert (BGE-M3 cosine) — sucht den
  passenden Absatz im existierenden Kundendokument fuer jeden Finding.
  Per-Run Thread-Local-Cache. Fallback: keyword-Match.
- Email-Rendering integriert Recipe + Anchor pro Doc-Pruefungs-Fail
  + Vendor-Flag-Liste mit aufklappbarer Action-Liste.
- Score-Erklaerung pro Vendor-Zeile (3/5-Untertitel + Tooltip).

Migration-Pipeline (Compliance-Check -> Customer Banner/Documents)
- migration_to_banner.py: Vendor-Liste -> CookieBannerConfig mit
  4 Kategorien + Review-Flags.
- migration_to_document.py: Vendor-Liste -> Cookie-Policy + VVT-Register
  + Privacy-Policy-Pre-Fills.
- agent_migration_routes: 3 Preview-Endpoints (banner-preview,
  document-preview, summary). Persistierung der cmp_vendors in
  /data/compliance_audits.db check_payloads-Tabelle.

Borlabs-Parity Cookie-Banner-Features
- Consent-Historie im Banner: window.bpShowConsentHistory() + localStorage.
- Content-Blocker: cookie-banner-content-blocker.ts — YouTube/Maps/Video
  Placeholder bis Einwilligung.
- Google Consent Mode v2 erweitert: wait_for_update + region=EEA/CH/GB.
- Consent-Log Export (CSV/JSON) per einwilligungen_export_routes.

Bug-Fixes
- canonical_control_routes: _jsonish-Helper fuer string-typed jsonb,
  similar-controls-Endpoint mit _has_embedding_col()-Cache (kein 500 mehr).
- Control-Library Frontend: defensive .map-Coercer in 2 Detail-Views.
- Embedding-Service-Batching (32er Batches statt 165 in einem Call).
- KeyError 'control_id' in MC-Result-Aggregation (defensive .get).
- Master-Controls-Klick-Through von /sdk/master-controls auf
  /sdk/control-library?control=<id> mit URL-Param-Auto-Open.
- Dockerfile: /data pre-chowned auf appuser (Audit-DB-Schreibrecht).
- Cookie-Text-Routing-Bug (cmp_reconstructed > DOM-extraction).
- doc_type-aware MC-Filter (statt all-text-MCs).
- Master-Contract-Dedup (60 BMW-Internal-Eintraege = 1 Adobe-Vertrag).
- A3-v2-Audit hat 24 UI-Sprache-MCs als 'process' reklassifiziert.

Tests
- test_migration_mappers.py (9 Tests)
- test_migration_endpoints.py (4 Tests)

Skripte (one-shot)
- classify_mc_check_type.py (v1) + _v2 (PK=control_id,doc_type)
- audit_mc_doctype_fit.py (v1 fits) + _v2 (ui_only + scope_requires)

BMW-Run-Bilanz v1 (broken) -> v9 (alle Fixes):
  DSE     7,5% -> 81-83%
  Impressum 4%   -> 100% (6 echte MCs alle erfuellt)
  Cookie  0%    -> 79-83% (CMP-Text-Routing + Embedding)
  Plus: 10 Konsolidierungs-Kategorien, geschaetzte Saving 200k-3M / Jahr
  Plus: Action-Recipes + Doc-Anchors fuer jeden Fail

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

backend-compliance/compliance/api/canonical_control_routes.py

@@ -1808,6 +1808,32 @@ async def list_categories():
 # SIMILAR CONTROLS (Embedding-based dedup)
 # =============================================================================
 
+_EMBEDDING_COL_AVAILABLE: bool | None = None
+
+
+def _has_embedding_col() -> bool:
+    """Cache whether canonical_controls has the embedding column.
+
+    Returns False on systems where pgvector + embedding backfill weren't
+    set up. Saves the per-request 500 + log spam.
+    """
+    global _EMBEDDING_COL_AVAILABLE
+    if _EMBEDDING_COL_AVAILABLE is not None:
+        return _EMBEDDING_COL_AVAILABLE
+    try:
+        with SessionLocal() as db:
+            r = db.execute(text(
+                "SELECT 1 FROM information_schema.columns "
+                "WHERE table_schema='compliance' "
+                "AND table_name='canonical_controls' "
+                "AND column_name='embedding'"
+            )).fetchone()
+            _EMBEDDING_COL_AVAILABLE = bool(r)
+    except Exception:
+        _EMBEDDING_COL_AVAILABLE = False
+    return _EMBEDDING_COL_AVAILABLE
+
+
 @router.get("/controls/{control_id}/similar")
 async def find_similar_controls(
     control_id: str,
@@ -1815,6 +1841,8 @@ async def find_similar_controls(
     limit: int = Query(20, ge=1, le=100),
 ):
     """Find controls similar to the given one using embedding cosine similarity."""
+    if not _has_embedding_col():
+        return []
     with SessionLocal() as db:
         # Get the target control's embedding
         target = db.execute(
@@ -1856,7 +1884,7 @@ async def find_similar_controls(
                     "title": r.title,
                     "severity": r.severity,
                     "release_state": r.release_state,
-                    "tags": r.tags or [],
+                    "tags": _jsonish(r.tags) or [],
                     "license_rule": r.license_rule,
                     "verification_method": r.verification_method,
                     "category": r.category,
@@ -1866,6 +1894,10 @@ async def find_similar_controls(
             ]
         except Exception as e:
             logger.warning("Embedding similarity query failed (no embedding column?): %s", e)
+            try:
+                db.rollback()
+            except Exception:
+                pass
             return []
 
 
@@ -1946,6 +1978,22 @@ async def get_v1_matches_endpoint(control_id: str):
 # INTERNAL HELPERS
 # =============================================================================
 
+def _jsonish(v):
+    """Parse v as JSON if it's a string that looks like JSON, otherwise return as-is.
+
+    Some canonical_controls rows were inserted with jsonb columns containing
+    raw JSON strings (e.g. '["a","b"]' as a TEXT). The frontend expects real
+    arrays — coerce here so .map() works.
+    """
+    if isinstance(v, str) and v and v[0] in "[{":
+        try:
+            import json as _j
+            return _j.loads(v)
+        except Exception:
+            return v
+    return v
+
+
 def _control_row(r) -> dict:
     return {
         "id": str(r.id),
@@ -1954,17 +2002,17 @@ def _control_row(r) -> dict:
         "title": r.title,
         "objective": r.objective,
         "rationale": r.rationale,
-        "scope": r.scope,
-        "requirements": r.requirements,
-        "test_procedure": r.test_procedure,
-        "evidence": r.evidence,
+        "scope": _jsonish(r.scope),
+        "requirements": _jsonish(r.requirements),
+        "test_procedure": _jsonish(r.test_procedure) or [],
+        "evidence": _jsonish(r.evidence) or [],
         "severity": r.severity,
         "risk_score": float(r.risk_score) if r.risk_score is not None else None,
         "implementation_effort": r.implementation_effort,
         "evidence_confidence": float(r.evidence_confidence) if r.evidence_confidence is not None else None,
-        "open_anchors": r.open_anchors,
+        "open_anchors": _jsonish(r.open_anchors) or [],
         "release_state": r.release_state,
-        "tags": r.tags or [],
+        "tags": _jsonish(r.tags) or [],
         "license_rule": r.license_rule,
         "source_original_text": r.source_original_text,
         "source_citation": r.source_citation,