feat(compliance-check): MC-Classification + Embedding + Vendor-Redundanz + Action-Recipes + Borlabs-Features

Massiv-Update auf Basis BMW-Test-Iterationen (v1→v9):

Core Compliance-Check
- Sonnet check_type Klassifikation: text/process/review fuer alle 1874 MCs
  in compliance.doc_check_controls (script + Sidecar /data/mc_classification.db).
  rag_document_checker filtert auf check_type='text' fuer doc_check.
  Plus fits_doc_type-Audit (v2) + ui_only-Audit fuer DSA/E-Commerce-MCs in
  falscher doc_type-Schublade.
- scope_requires-Filter: biometric/ai_decision/child_targeting MCs werden
  per business_profile gefiltert (FRT skipped fuer BMW etc.).
- Embedding-Match (BGE-M3) als Phase-3 nach Regex-Match:
  Per-doc_type-Threshold-Override (impressum 0.50, dse/cookie 0.60),
  Short-Field-Rescue (15-Wort-Chunks) fuer Pflichtfelder im Impressum.
  Title+check_question als Embedding-Input fuer mehr Kontext.
- Cookie-Text-Routing: consent-tester gibt cmp_cookie_text aus dem
  CMP-Reconstruct zurueck, Backend bevorzugt das gegen DOM-Extraction
  wenn richer (BMW 1824 vs 600 Worte).

Vendor-Redundanz + EU-Alternativen + Cost-Saving
- vendor_redundancy.analyze() — funktionale Kategorisierung der CMP-Vendors,
  Detektion von Mehrfach-Anbietern pro Kategorie, EU-Alternative-Lookup
  (Matomo, IONOS, HERE, Friendly Captcha, Smart AdServer, ...).
- vendor_cost_estimator: Tier-Inferenz aus Cookie-Footprint (Cookie-Anzahl
  + Premium-Feature-Cookies + Third-Party-Quote → starter/professional/
  enterprise/premier).
- Self-Service-Werbung (Google/Meta/Pinterest/...) = 0 Lizenz-Kosten
  (nur Media-Spend, separat). DSP-Plattformen behalten enge Range.
- Tier-aware Saving-Range: bei Enterprise/Premier nutzen wir den
  oberen 40-100%-Band der Listpreise, nicht starter→premier.
- Multi-Function-Tools (Matomo Pro, SAP CX, IONOS Cloud, Userlike, Smart
  AdServer, HERE Maps, Vimeo Pro, LamaPoll) — ein Tool ersetzt mehrere
  Kategorien gleichzeitig.

Cookie-Wissens-DB + Funktionale Klassifikation
- cookie_knowledge_db: 50 kuratierte Top-Cookies (Google/Meta/Adobe/MS/...)
  mit vendor, exact_purpose, data_collected, IAB-TCF-IDs, reid_risk,
  schrems_ii_status, EuGH-Urteile, EU-Alternative.
- cookie_function_classifier: pro Cookie funktionale Rolle (tracking_id,
  ad_pixel, session_id, ab_test, csrf, ...) + blocking_impact.

Country-Inferenz aus Rechtsform
- cookie_link_validator: Country-Field wird aus Vendor-Name abgeleitet
  (A/S=DK, GmbH=DE, Inc=US, B.V.=NL, ...) plus Vendor-Lookup-Table.
  Reduziert false-positive no_country-Flags bei eindeutig-EU-Vendors
  (Adform DK, Pinterest IE).

Action-Recipes + Doc-Anchor-Locator
- finding_action_recipes: pro Finding-Typ (no_cookies_listed, no_country,
  broken_opt_out, "Auftragsverarbeiter erwaehnen", "Art. 22 Profiling",
  ...) eine strukturierte Anweisung mit what/why/fix_text/where/example.
  Zum 1:1-Einfuegen in Kunden-Dokumente.
- doc_anchor_locator: Embedding-basiert (BGE-M3 cosine) — sucht den
  passenden Absatz im existierenden Kundendokument fuer jeden Finding.
  Per-Run Thread-Local-Cache. Fallback: keyword-Match.
- Email-Rendering integriert Recipe + Anchor pro Doc-Pruefungs-Fail
  + Vendor-Flag-Liste mit aufklappbarer Action-Liste.
- Score-Erklaerung pro Vendor-Zeile (3/5-Untertitel + Tooltip).

Migration-Pipeline (Compliance-Check -> Customer Banner/Documents)
- migration_to_banner.py: Vendor-Liste -> CookieBannerConfig mit
  4 Kategorien + Review-Flags.
- migration_to_document.py: Vendor-Liste -> Cookie-Policy + VVT-Register
  + Privacy-Policy-Pre-Fills.
- agent_migration_routes: 3 Preview-Endpoints (banner-preview,
  document-preview, summary). Persistierung der cmp_vendors in
  /data/compliance_audits.db check_payloads-Tabelle.

Borlabs-Parity Cookie-Banner-Features
- Consent-Historie im Banner: window.bpShowConsentHistory() + localStorage.
- Content-Blocker: cookie-banner-content-blocker.ts — YouTube/Maps/Video
  Placeholder bis Einwilligung.
- Google Consent Mode v2 erweitert: wait_for_update + region=EEA/CH/GB.
- Consent-Log Export (CSV/JSON) per einwilligungen_export_routes.

Bug-Fixes
- canonical_control_routes: _jsonish-Helper fuer string-typed jsonb,
  similar-controls-Endpoint mit _has_embedding_col()-Cache (kein 500 mehr).
- Control-Library Frontend: defensive .map-Coercer in 2 Detail-Views.
- Embedding-Service-Batching (32er Batches statt 165 in einem Call).
- KeyError 'control_id' in MC-Result-Aggregation (defensive .get).
- Master-Controls-Klick-Through von /sdk/master-controls auf
  /sdk/control-library?control=<id> mit URL-Param-Auto-Open.
- Dockerfile: /data pre-chowned auf appuser (Audit-DB-Schreibrecht).
- Cookie-Text-Routing-Bug (cmp_reconstructed > DOM-extraction).
- doc_type-aware MC-Filter (statt all-text-MCs).
- Master-Contract-Dedup (60 BMW-Internal-Eintraege = 1 Adobe-Vertrag).
- A3-v2-Audit hat 24 UI-Sprache-MCs als 'process' reklassifiziert.

Tests
- test_migration_mappers.py (9 Tests)
- test_migration_endpoints.py (4 Tests)

Skripte (one-shot)
- classify_mc_check_type.py (v1) + _v2 (PK=control_id,doc_type)
- audit_mc_doctype_fit.py (v1 fits) + _v2 (ui_only + scope_requires)

BMW-Run-Bilanz v1 (broken) -> v9 (alle Fixes):
  DSE     7,5% -> 81-83%
  Impressum 4%   -> 100% (6 echte MCs alle erfuellt)
  Cookie  0%    -> 79-83% (CMP-Text-Routing + Embedding)
  Plus: 10 Konsolidierungs-Kategorien, geschaetzte Saving 200k-3M / Jahr
  Plus: Action-Recipes + Doc-Anchors fuer jeden Fail

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

backend-compliance/compliance/services/rag_document_checker.py

@@ -37,6 +37,7 @@ async def check_document_with_controls(
     db_url: str = "",
     max_controls: int = 0,  # 0 = no limit, check ALL
     use_agent: bool = False,  # Use LLM agent for intelligent evaluation
+    business_scope: set[str] | None = None,
 ) -> list[dict]:
     """Check document against ALL doc_check_controls for this doc_type.
 
@@ -56,7 +57,7 @@ async def check_document_with_controls(
     mapped_type = _map_doc_type(doc_type)
 
     # Load ALL controls for this doc_type
-    controls = await _load_controls(mapped_type, db_url, max_controls)
+    controls = await _load_controls(mapped_type, db_url, max_controls, business_scope)
     if not controls:
         logger.info("No MCs for doc_type '%s' (%s)", mapped_type, doc_title)
         return []
@@ -71,6 +72,31 @@ async def check_document_with_controls(
         if result:
             results.append(result)
 
+    # Semantic fallback (Phase 3): MCs that failed via regex get a second
+    # chance via BGE-M3 cosine similarity. BMW writes "Speicherdauer 2
+    # Jahre" — the regex misses, embedding catches it.
+    failed_ids = {r.get("control_id") for r in results
+                  if not r.get("passed") and r.get("control_id")}
+    if failed_ids:
+        try:
+            from compliance.services.mc_embedding_matcher import (
+                ensure_mc_embeddings, embedding_match,
+            )
+            await ensure_mc_embeddings()  # idempotent: only embeds new MCs
+            failed_mcs = [c for c in controls if c.get("control_id") in failed_ids]
+            semantic_passes = await embedding_match(
+                text, failed_mcs, doc_type=mapped_type,
+            )
+            if semantic_passes:
+                for r in results:
+                    cid = r.get("control_id")
+                    if cid and cid in semantic_passes and not r.get("passed"):
+                        r["passed"] = True
+                        r["matched_text"] = "[semantischer Treffer via Embedding]"
+                        r["hint"] = (r.get("hint") or "") + " (passed via Embedding-Match, BGE-M3 cosine)"
+        except Exception as e:
+            logger.warning("Embedding fallback skipped: %s", e, exc_info=True)
+
     passed = sum(1 for r in results if r["passed"])
     failed_results = [r for r in results if not r["passed"]]
     logger.info("MC results: %d passed, %d failed out of %d for '%s'",
@@ -161,6 +187,7 @@ def _check_mc_deterministic(text_lower: str, mc: dict) -> Optional[dict]:
 
     return {
         "id": f"mc-{control_id}",
+        "control_id": control_id,
         "label": mc.get("title", "")[:80],
         "passed": passed,
         "severity": severity,
@@ -266,11 +293,72 @@ _MC_ALIAS_FALLBACK = {
 }
 
 
-async def _load_controls(doc_type: str, db_url: str, limit: int) -> list[dict]:
+def _load_text_only_ids(
+    doc_type: str | None = None,
+    business_scope: set[str] | None = None,
+) -> set[str]:
+    """Return control_ids that the Sonnet-classifier flagged as 'text'.
+
+    Filters applied:
+    1. check_type='text' (only doc-text-matchable MCs)
+    2. doc_type matches (per-doc-type variant from v2-Sidecar)
+    3. fits_doc_type=1 (LLM auditor approved this MC for this doc_type)
+    4. scope_requires NULL or contained in business_scope
+       (e.g. MCs with scope_requires='biometric_processing' are skipped
+       on sites that don't do biometric processing — Art. 22 FRT-MC bei
+       BMW falsch-positiv)
+
+    `business_scope` comes from the business_profiler (set of detected
+    site characteristics like 'b2c', 'shop', 'biometric_processing',
+    'ai_decision_making', 'child_targeting').
+
+    Returns empty set if the sidecar doesn't exist yet.
+    """
+    import sqlite3
+    db_path = os.getenv("MC_CLASS_DB", "/data/mc_classification.db")
+    try:
+        with sqlite3.connect(db_path) as c:
+            cols = [r[1] for r in c.execute("PRAGMA table_info(mc_classification)")]
+            has_fit = "fits_doc_type" in cols
+            has_scope = "scope_requires" in cols
+            fit_clause = " AND (fits_doc_type IS NULL OR fits_doc_type = 1)" if has_fit else ""
+            base = ("SELECT control_id, scope_requires FROM mc_classification "
+                    "WHERE check_type = 'text'" + fit_clause) if has_scope else (
+                   "SELECT control_id, NULL FROM mc_classification "
+                   "WHERE check_type = 'text'" + fit_clause)
+            params: list = []
+            if doc_type:
+                base += " AND doc_type = ?"
+                params.append(doc_type)
+            rows = c.execute(base, params).fetchall()
+            scope = business_scope or set()
+            keep: set[str] = set()
+            for cid, req in rows:
+                if not req:
+                    keep.add(cid)
+                else:
+                    # Multiple requirements separated by '|' — ALL must
+                    # be in scope to include. Empty req tokens are skipped.
+                    needed = {r.strip().lower() for r in req.split("|") if r.strip()}
+                    if needed.issubset({s.lower() for s in scope}):
+                        keep.add(cid)
+            return keep
+    except sqlite3.OperationalError:
+        return set()
+    except Exception as e:
+        logger.warning("MC classification lookup failed: %s", e)
+        return set()
+
+
+async def _load_controls(doc_type: str, db_url: str, limit: int,
+                         business_scope: set[str] | None = None) -> list[dict]:
     """Load all doc_check_controls for a doc_type from PostgreSQL.
 
     Falls back via _MC_ALIAS_FALLBACK when no MCs exist for the requested
     type (e.g. 'nutzungsbedingungen' -> 'agb').
+
+    Filters to only check_type='text' MCs when the classification sidecar
+    is present — process/review MCs are routed to other modules.
     """
     try:
         import asyncpg
@@ -297,7 +385,17 @@ async def _load_controls(doc_type: str, db_url: str, limit: int) -> list[dict]:
             fallback = _MC_ALIAS_FALLBACK[doc_type]
             logger.info("No MCs for %s -> falling back to %s", doc_type, fallback)
             rows = await conn.fetch(query, fallback)
-        return [dict(r) for r in rows]
+
+        controls = [dict(r) for r in rows]
+        text_only = _load_text_only_ids(doc_type, business_scope)
+        if text_only:
+            before = len(controls)
+            controls = [c for c in controls if c.get("control_id") in text_only]
+            logger.info(
+                "MC filter (text only) for %s: %d/%d MCs after Sonnet check_type filter",
+                doc_type, len(controls), before,
+            )
+        return controls
     except Exception as e:
         logger.warning("MC query failed: %s", e)
         return []