Benjamin_Boenisch/breakpilot-compliance
1
0
Fork 0
Code Issues 24 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(compliance-check): MC-Classification + Embedding + Vendor-Redundanz + Action-Recipes + Borlabs-Features
CI / nodejs-build (push) Successful in 2m47s
Details
CI / branch-name (push) Has been skipped
Details
CI / guardrail-integrity (push) Has been skipped
Details
CI / detect-changes (push) Successful in 10s
Details
CI / secret-scan (push) Has been skipped
Details
CI / dep-audit (push) Has been skipped
Details
CI / sbom-scan (push) Has been skipped
Details
CI / validate-canonical-controls (push) Successful in 16s
Details
CI / loc-budget (push) Failing after 17s
Details
CI / go-lint (push) Has been skipped
Details
CI / python-lint (push) Has been skipped
Details
CI / nodejs-lint (push) Has been skipped
Details
CI / test-python-backend (push) Successful in 42s
Details
CI / test-python-document-crawler (push) Has been skipped
Details
CI / test-go (push) Has been skipped
Details
CI / iace-gt-coverage (push) Has been skipped
Details
CI / test-python-dsms-gateway (push) Has been skipped
Details

Browse Source 
Massiv-Update auf Basis BMW-Test-Iterationen (v1→v9):

Core Compliance-Check
- Sonnet check_type Klassifikation: text/process/review fuer alle 1874 MCs
  in compliance.doc_check_controls (script + Sidecar /data/mc_classification.db).
  rag_document_checker filtert auf check_type='text' fuer doc_check.
  Plus fits_doc_type-Audit (v2) + ui_only-Audit fuer DSA/E-Commerce-MCs in
  falscher doc_type-Schublade.
- scope_requires-Filter: biometric/ai_decision/child_targeting MCs werden
  per business_profile gefiltert (FRT skipped fuer BMW etc.).
- Embedding-Match (BGE-M3) als Phase-3 nach Regex-Match:
  Per-doc_type-Threshold-Override (impressum 0.50, dse/cookie 0.60),
  Short-Field-Rescue (15-Wort-Chunks) fuer Pflichtfelder im Impressum.
  Title+check_question als Embedding-Input fuer mehr Kontext.
- Cookie-Text-Routing: consent-tester gibt cmp_cookie_text aus dem
  CMP-Reconstruct zurueck, Backend bevorzugt das gegen DOM-Extraction
  wenn richer (BMW 1824 vs 600 Worte).

Vendor-Redundanz + EU-Alternativen + Cost-Saving
- vendor_redundancy.analyze() — funktionale Kategorisierung der CMP-Vendors,
  Detektion von Mehrfach-Anbietern pro Kategorie, EU-Alternative-Lookup
  (Matomo, IONOS, HERE, Friendly Captcha, Smart AdServer, ...).
- vendor_cost_estimator: Tier-Inferenz aus Cookie-Footprint (Cookie-Anzahl
  + Premium-Feature-Cookies + Third-Party-Quote → starter/professional/
  enterprise/premier).
- Self-Service-Werbung (Google/Meta/Pinterest/...) = 0 Lizenz-Kosten
  (nur Media-Spend, separat). DSP-Plattformen behalten enge Range.
- Tier-aware Saving-Range: bei Enterprise/Premier nutzen wir den
  oberen 40-100%-Band der Listpreise, nicht starter→premier.
- Multi-Function-Tools (Matomo Pro, SAP CX, IONOS Cloud, Userlike, Smart
  AdServer, HERE Maps, Vimeo Pro, LamaPoll) — ein Tool ersetzt mehrere
  Kategorien gleichzeitig.

Cookie-Wissens-DB + Funktionale Klassifikation
- cookie_knowledge_db: 50 kuratierte Top-Cookies (Google/Meta/Adobe/MS/...)
  mit vendor, exact_purpose, data_collected, IAB-TCF-IDs, reid_risk,
  schrems_ii_status, EuGH-Urteile, EU-Alternative.
- cookie_function_classifier: pro Cookie funktionale Rolle (tracking_id,
  ad_pixel, session_id, ab_test, csrf, ...) + blocking_impact.

Country-Inferenz aus Rechtsform
- cookie_link_validator: Country-Field wird aus Vendor-Name abgeleitet
  (A/S=DK, GmbH=DE, Inc=US, B.V.=NL, ...) plus Vendor-Lookup-Table.
  Reduziert false-positive no_country-Flags bei eindeutig-EU-Vendors
  (Adform DK, Pinterest IE).

Action-Recipes + Doc-Anchor-Locator
- finding_action_recipes: pro Finding-Typ (no_cookies_listed, no_country,
  broken_opt_out, "Auftragsverarbeiter erwaehnen", "Art. 22 Profiling",
  ...) eine strukturierte Anweisung mit what/why/fix_text/where/example.
  Zum 1:1-Einfuegen in Kunden-Dokumente.
- doc_anchor_locator: Embedding-basiert (BGE-M3 cosine) — sucht den
  passenden Absatz im existierenden Kundendokument fuer jeden Finding.
  Per-Run Thread-Local-Cache. Fallback: keyword-Match.
- Email-Rendering integriert Recipe + Anchor pro Doc-Pruefungs-Fail
  + Vendor-Flag-Liste mit aufklappbarer Action-Liste.
- Score-Erklaerung pro Vendor-Zeile (3/5-Untertitel + Tooltip).

Migration-Pipeline (Compliance-Check -> Customer Banner/Documents)
- migration_to_banner.py: Vendor-Liste -> CookieBannerConfig mit
  4 Kategorien + Review-Flags.
- migration_to_document.py: Vendor-Liste -> Cookie-Policy + VVT-Register
  + Privacy-Policy-Pre-Fills.
- agent_migration_routes: 3 Preview-Endpoints (banner-preview,
  document-preview, summary). Persistierung der cmp_vendors in
  /data/compliance_audits.db check_payloads-Tabelle.

Borlabs-Parity Cookie-Banner-Features
- Consent-Historie im Banner: window.bpShowConsentHistory() + localStorage.
- Content-Blocker: cookie-banner-content-blocker.ts — YouTube/Maps/Video
  Placeholder bis Einwilligung.
- Google Consent Mode v2 erweitert: wait_for_update + region=EEA/CH/GB.
- Consent-Log Export (CSV/JSON) per einwilligungen_export_routes.

Bug-Fixes
- canonical_control_routes: _jsonish-Helper fuer string-typed jsonb,
  similar-controls-Endpoint mit _has_embedding_col()-Cache (kein 500 mehr).
- Control-Library Frontend: defensive .map-Coercer in 2 Detail-Views.
- Embedding-Service-Batching (32er Batches statt 165 in einem Call).
- KeyError 'control_id' in MC-Result-Aggregation (defensive .get).
- Master-Controls-Klick-Through von /sdk/master-controls auf
  /sdk/control-library?control=<id> mit URL-Param-Auto-Open.
- Dockerfile: /data pre-chowned auf appuser (Audit-DB-Schreibrecht).
- Cookie-Text-Routing-Bug (cmp_reconstructed > DOM-extraction).
- doc_type-aware MC-Filter (statt all-text-MCs).
- Master-Contract-Dedup (60 BMW-Internal-Eintraege = 1 Adobe-Vertrag).
- A3-v2-Audit hat 24 UI-Sprache-MCs als 'process' reklassifiziert.

Tests
- test_migration_mappers.py (9 Tests)
- test_migration_endpoints.py (4 Tests)

Skripte (one-shot)
- classify_mc_check_type.py (v1) + _v2 (PK=control_id,doc_type)
- audit_mc_doctype_fit.py (v1 fits) + _v2 (ui_only + scope_requires)

BMW-Run-Bilanz v1 (broken) -> v9 (alle Fixes):
  DSE     7,5% -> 81-83%
  Impressum 4%   -> 100% (6 echte MCs alle erfuellt)
  Cookie  0%    -> 79-83% (CMP-Text-Routing + Embedding)
  Plus: 10 Konsolidierungs-Kategorien, geschaetzte Saving 200k-3M / Jahr
  Plus: Action-Recipes + Doc-Anchors fuer jeden Fail

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Benjamin Admin
2026-05-18 18:30:08 +02:00
parent 52fb8b91e7
commit 662327e8b4
31 changed files with 5214 additions and 104 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend-compliance/compliance/services/cookie_link_validator.py
+156
View File
@@ -220,10 +220,16 @@ def score_vendors(vendors: list[dict]) -> list[dict]:
flags.append("no_purpose")
# Country — only for external processors / controllers
# Falls country leer ist, ableiten aus Rechtsform-Suffix im Namen.
if country_required:
max_score += 10
if v.get("country"):
score += 10
elif _country_from_name(v.get("name", "")):
inferred = _country_from_name(v.get("name", ""))
v["country"] = inferred
v["country_inferred"] = True
score += 10
else:
flags.append("no_country")
@@ -321,3 +327,153 @@ def build_check_items(validated: list[LinkCheck]) -> list[dict]:
"hint": hint,
})
return items
# ─── Country-Inferenz aus Rechtsform-Suffix ────────────────────────
#
# Wenn ein Vendor das "country"-Feld leer hat, koennen wir es oft aus
# dem Firmen-Suffix ableiten:
# Adform A/S → DK (Dänemark, Aktieselskab)
# Pinterest Europe Ltd. → IE (Irland, Limited)
# Salesforce Inc. → US (Incorporated)
# Adobe ... Ireland Limited → IE
# Genesys ... B.V. → NL (Niederlande, Besloten Vennootschap)
# Equativ S.A. → FR (Société Anonyme)
# SAP SE → DE (Societas Europaea — meist DE-eingetragen)
#
# Kombi-Strategie:
# 1) Suffix-Pattern
# 2) Laendername im Firmen-Namen ('Ireland', 'Deutschland')
# 3) Specific Vendor (Google Inc / Meta Platforms Ireland Ltd → vendor-specific)
import re as _re
_SUFFIX_COUNTRY: list[tuple[str, str]] = [
# Pattern (am Wort-Ende oder vor weiteren Tokens) → ISO-Code
(r"\bA/S\b", "DK"), # Aktieselskab
(r"\bApS\b", "DK"), # Anpartsselskab
(r"\bAB\b", "SE"), # Aktiebolag
(r"\bAS\b(?!\w)", "NO"), # Aksjeselskap
(r"\bOy\b", "FI"), # Osakeyhtiö
(r"\bAG\b(?!\w)", "DE"), # auch CH/AT moeglich, default DE
(r"\bGmbH\b", "DE"),
(r"\bUG\b", "DE"),
(r"\beG\b", "DE"),
(r"\bKG\b", "DE"),
(r"\bOHG\b", "DE"),
(r"\bSE\b", "DE"), # Societas Europaea — pruefen ob SAP SE etc.
(r"\bS\.A\.\b", "FR"), # France / SE / ES
(r"\bSAS\b", "FR"),
(r"\bS\.A\.S\.\b", "FR"),
(r"\bSARL\b", "FR"),
(r"\bS\.r\.l\.\b", "IT"),
(r"\bS\.p\.A\.\b", "IT"),
(r"\bSpA\b", "IT"),
(r"\bB\.V\.\b", "NL"),
(r"\bN\.V\.\b", "NL"),
(r"\bSL\b", "ES"),
(r"\bS\.A\.\sde C\.V\.\b", "MX"),
(r"\bd\.o\.o\.\b", "SI"), # Slowenien
(r"\bd\.d\.\b", "HR"), # Kroatien
(r"\bz\s?o\.o\.\b", "PL"),
(r"\bInc\.?\b", "US"),
(r"\bIncorporated\b", "US"),
(r"\bCorp\.?\b", "US"),
(r"\bCorporation\b", "US"),
(r"\bLLC\b", "US"),
(r"\bL\.L\.C\.\b", "US"),
(r"\bLtd\.?\b", "GB"), # UK Limited, default
(r"\bLimited\b", "GB"),
(r"\bPLC\b", "GB"),
(r"\bPty\b", "AU"),
(r"\bK\.K\.\b", "JP"), # Kabushiki-Kaisha
(r"\bPte\.?\sLtd\.?\b", "SG"),
]
# Country-Namen im Firmen-Namen (z.B. "Adobe Systems Software Ireland Limited")
_COUNTRY_NAME_TOKENS: list[tuple[str, str]] = [
("ireland", "IE"),
("deutschland", "DE"),
("germany", "DE"),
("netherlands", "NL"),
("france", "FR"),
("united kingdom", "GB"),
("uk", "GB"),
("usa", "US"),
("united states", "US"),
("austria", "AT"),
("oesterreich", "AT"),
("schweiz", "CH"),
("switzerland", "CH"),
("luxembourg", "LU"),
("luxemburg", "LU"),
("denmark", "DK"),
("daenemark", "DK"),
("sweden", "SE"),
("schweden", "SE"),
("norway", "NO"),
("norwegen", "NO"),
("finland", "FI"),
("finnland", "FI"),
]
# Bekannte Vendors mit eindeutigem Sitz (override)
_KNOWN_VENDOR_COUNTRY: dict[str, str] = {
"google inc": "US",
"google llc": "US",
"google ireland": "IE",
"meta platforms ireland": "IE",
"facebook ireland": "IE",
"amazon.com inc": "US",
"amazon web services": "US",
"amazon web services inc": "US",
"linkedin inc": "US",
"salesforce inc": "US",
"salesforce.com": "US",
"outbrain inc": "US",
"taboola inc": "US",
"pinterest europe ltd": "IE",
"intuition machines inc": "US",
"akamai technologies inc": "US",
"criteo s.a": "FR",
"criteo sa": "FR",
"adform a/s": "DK",
"speedcurve limited": "GB",
"longtail ad solutions": "US",
"genesys cloud services b.v": "NL",
"qualtrics": "US",
"teads sa": "FR",
"teads s.a": "FR",
"salesviewer gmbh": "DE",
"baqend gmbh": "DE",
"zenweshare sas": "FR",
"nayoki gmbh": "DE",
"psyma": "DE",
"matomo": "NZ", # InnoCraft NZ aber EU-hostbar
"adobe systems software ireland": "IE",
"microsoft corporation": "US",
"microsoft corp": "US",
}
def _country_from_name(vendor_name: str) -> str:
"""Best-effort: ISO-2 Country-Code aus dem Vendor-Namen ableiten."""
if not vendor_name:
return ""
# Vendor-Namen sind oft "<Firma> — <Tool>" — nur Firmen-Teil betrachten
firm = vendor_name.split(" — ")[0].strip()
firm_l = firm.lower()
# 1) Known vendor lookup (most specific)
for k, v in _KNOWN_VENDOR_COUNTRY.items():
if k in firm_l:
return v
# 2) Country-Name im Firmen-Namen
for token, code in _COUNTRY_NAME_TOKENS:
if token in firm_l:
return code
# 3) Rechtsform-Suffix
for pattern, code in _SUFFIX_COUNTRY:
if _re.search(pattern, firm):
return code
return ""