Files
breakpilot-compliance/backend-compliance/compliance/api/agent_check/_b3_wiring.py
T
Benjamin Admin d0e3621192 feat(audit): V2 mail render + 5 new findings (B4/B5/B6/B7/B8) + LLM-Plausibility-Phase
Mail Render V2 (compliance/services/mail_render_v2/) — 11-Modul-Subpackage
das einen einheitlichen Audit-Mail-Output erzeugt mit:
  - Header + KPI-Kacheln (Score / Findings / Docs / Vendors)
  - TOC + Sprung-Links
  - 3-Bucket-Trennung: Kritische Befunde / Manuelle Prüfung / Interne Reminder
  - Cookie-Inventar (Name·Vendor·Kategorie·Speicherdauer·Löschfrist·Sitzland·Quelle·Status)
  - Sofortmaßnahmen-Aggregator ("Sitzland ergänzen für 11 Cookies")
  - 24 Legacy-Wrappers — alle alten build_*_html in V2-Sections
  - Scope-Filter: FIN/GOV/MED/INS/EDU/LEG aus Berichten wenn nicht relevant
  - Hint/Action-Dedup: keine doppelten Sätze pro Card mehr
Aktiviert via env MAIL_RENDER_V2=true (Default: legacy renderer).

5 neue deterministische Findings als Phase D-2b/B4/B5/B6/B7/B8:

  B4 vendor_consistency_check — Cross-Doc-Provider-Widerspruch
     (Elli: DSE nennt Vertex AI für Chatbot, /de/cookies nennt Iadvize → HIGH).
     6 Service-Types: chatbot/analytics/tag_manager/pixel/cdn/cmp.

  B5 ai_act_transparency_check — AI Act Art. 50 Transparenzpflicht
     (Elli: Vertex AI vorhanden ohne Pre-Chat-Disclosure → HIGH).
     Plus B5-Erweiterung: Rechtsgrundlage Art-6-Abs-1-lit-f bei AI → MED
     (Einwilligung empfehlen).

  B6 cross_doc_dpo_check — DPO in DSE genannt, nicht im Impressum (LOW).

  B7 doc_staleness_check — Datum-Extraktion aus DSE/AGB/Nutzungsbedingungen.
     Cap: AGB/NB 3y, DSE 2y. Älter → MEDIUM (Elli NB Stand 2018 → HIGH).

  B8 cmp_fingerprint_check — Banner detected, aber CMP-Provider generic
     (kein Usercentrics/OneTrust/Cookiebot/etc → MED).

  B3-Erweiterung detect_intra_doc_contradictions — Widersprüchliche
     Speicherdauer im SELBEN Doc (Elli: Logfile 7d vs 30d → HIGH).

LLM-Plausibility-Phase (Phase D-2b, finding_plausibility_check.py):
  - Läuft AFTER MC pipeline, BEFORE D3 render
  - Prompt mit Beispiel-IDs + 3-Phase-Mapping: exact-ID / position-fallback /
    fuzzy-tail-match
  - Stempelt llm_title / llm_severity / llm_recommendation / llm_drop auf
    jeden FAIL CheckItem
  - V2-Render zeigt "🤖 LLM-Plausibility:" Box pro Finding wenn gestempelt
  - KNOWN ISSUE: qwen3:30b-a3b liefert oft empty content auf format='json' +
    8000-char-excerpt prompts. Pipeline läuft mit stamped=0 weiter. Task #16.

Coverage gegen Elli Ground Truth (zeroclaw/docs/ground-truth/elli_eco_2026-06-06.json,
13 expected findings via WebFetch-Agent-Crawl):
  - 4/4 HIGH-Findings ✓ (COOKIE-CONSENT-UX-001 + WIDERRUFSBELEHRUNG-001 +
    VENDOR-CONSISTENCY-001 + AI-ACT-TRANSPARENCY-001)
  - 4/6 MEDIUM ✓
  - 2/3 LOW ✓
  - Total: 10/13 = 77% (Sprung von 4/13 = 31%)

Restliche 3 Gaps als Task #17: IMPRESSUM-001 (multi-entity USt-IdNr),
TRANSFER-001 (Vendor-Mechanismus DPF/SCC), TH-RETENTION-002 (AI-Retention
pro Datenkategorie).

V2-Mail-Preview in Mailpit: 'v2all@local.test' Subject '[V2 ALL] ELLI'.
Backend healthy, B1+B3+B4+B5+B6+B7+B8 alle live im Orchestrator.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-06 21:19:49 +02:00

196 lines
7.1 KiB
Python

"""B3 wiring — Cross-doc retention consistency check + HTML block.
Combines three sources of retention truth per cookie:
- DSI text (state["doc_texts"]["dse"] or "cookie")
- cookie-table `duration` from cmp_vendors[i]["cookies"][j]
- actual cookie expiry from banner_result["cookies_detailed"][k]
and produces per-cookie findings + a TH-RETENTION theme summary. Only
renders an HTML block when there are findings to show; the block is
sorted by severity (HIGH first) and shows the top-10 mismatches.
"""
from __future__ import annotations
import html
import logging
import time
from compliance.services.retention_comparator import (
build_retention_theme_summary,
compare_retention,
detect_intra_doc_contradictions,
extract_retention_claims,
)
logger = logging.getLogger(__name__)
def _actual_max_age_seconds(cookie: dict) -> float | None:
"""Get cookie Max-Age in seconds.
Playwright gives us `expires` as a Unix timestamp (seconds-since-
epoch). Some sources give `max_age` directly. -1 / 0 means session
cookie (no expiry) — return None to signal that.
"""
ma = cookie.get("max_age")
if isinstance(ma, (int, float)) and ma > 0:
return float(ma)
exp = cookie.get("expires")
if isinstance(exp, (int, float)) and exp > 0:
delta = exp - time.time()
if delta > 0:
return float(delta)
return None
def run_b3(state: dict) -> None:
"""Cross-doc retention check + render HTML. Mutates state in place."""
doc_texts = state["doc_texts"]
cmp_vendors = state["cmp_vendors"]
banner_result = state["banner_result"]
dsi_text = doc_texts.get("dse") or doc_texts.get("cookie") or ""
if not dsi_text:
return
# Intra-doc contradictions are independent of cmp_vendors — run
# them first so they survive the early-return below.
intra = detect_intra_doc_contradictions(dsi_text)
state["retention_intra_doc"] = intra
cookie_records: list[dict] = []
cookie_names: list[str] = []
vendor_names: list[str] = []
for v in cmp_vendors or []:
vname = (v.get("name") or "").strip()
if vname:
vendor_names.append(vname)
for c in (v.get("cookies") or []):
cname = (c.get("name") or "").strip()
if not cname:
continue
duration = (c.get("duration") or c.get("persistence")
or c.get("expiry") or "")
cookie_names.append(cname)
cookie_records.append({
"name": cname,
"vendor": vname,
"table_duration": duration,
"actual_max_age": None,
})
if not cookie_records:
return
# Match actual max_age from banner_result.cookies_detailed
if banner_result:
cookies_detailed = banner_result.get("cookies_detailed") or []
by_name: dict[str, dict] = {}
for c in cookies_detailed:
n = (c.get("name") or "").lower()
if n:
by_name[n] = c
for rec in cookie_records:
nm = rec["name"].lower()
if nm in by_name:
rec["actual_max_age"] = _actual_max_age_seconds(by_name[nm])
claims = extract_retention_claims(dsi_text, cookie_names, vendor_names)
findings: list[dict] = []
for rec in cookie_records:
finding = compare_retention(
cookie_name=rec["name"],
table_duration=rec["table_duration"],
actual_max_age_seconds=rec["actual_max_age"],
dsi_claims=claims,
vendor_name=rec["vendor"] or None,
)
findings.append(finding)
summary = build_retention_theme_summary(findings)
state["retention_findings"] = findings
state["retention_theme_summary"] = summary
state["retention_html"] = _render_block(summary, findings)
logger.info(
"B3 Retention: %d findings, %d passed, %d failed, %d incomplete",
summary["total"], summary["passed"], summary["failed"],
summary["incomplete"],
)
def _fmt_days(d: float | None) -> str:
if d is None:
return ""
if d < 1:
return f"{int(d * 24)}h"
if d < 30:
return f"{int(d)}d"
if d < 365:
return f"{int(d / 30)}mo"
return f"{d / 365:.1f}y"
def _render_block(summary: dict, findings: list[dict]) -> str:
if summary["total"] == 0:
return ""
failed_findings = [f for f in findings if not f.get("matches")
and f.get("severity_reason") != "incomplete"]
if not failed_findings:
return "" # all OK, no block needed
# Sort by severity (HIGH first) then diff_days desc
sev_rank = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
failed_findings.sort(key=lambda f: (
sev_rank.get((f.get("severity") or "").upper(), 9),
-(f.get("diff_days") or 0),
))
rows = []
for f in failed_findings[:10]:
sev = (f.get("severity") or "").upper()
color = ("#dc2626" if sev == "HIGH"
else "#f59e0b" if sev == "MEDIUM" else "#64748b")
rows.append(
"<tr>"
f"<td style='padding:6px 10px;border-bottom:1px solid #e5e7eb;'>"
f"<code>{html.escape(f.get('cookie_name') or '')}</code></td>"
f"<td style='padding:6px 10px;border-bottom:1px solid #e5e7eb;'>"
f"{html.escape((f.get('vendor_name') or ''))}</td>"
f"<td style='padding:6px 10px;border-bottom:1px solid #e5e7eb;'>"
f"DSI: {_fmt_days(f.get('dsi_days'))}"
f"Tabelle: {_fmt_days(f.get('table_days'))}"
f"Realität: {_fmt_days(f.get('actual_days'))}</td>"
f"<td style='padding:6px 10px;border-bottom:1px solid #e5e7eb;"
f"color:{color};font-weight:600;'>"
f"{sev} ({html.escape(f.get('mismatch_type') or '')})</td>"
"</tr>"
)
total = summary["total"]
passed = summary["passed"]
failed = summary["failed"]
incomplete = summary["incomplete"]
return (
"<div style='margin:24px 0;padding:16px;border-left:4px solid #dc2626;"
"background:#fefce8;border-radius:4px;'>"
"<h2 style='margin:0 0 8px;color:#854d0e;font-size:16px;'>"
"TH-RETENTION — Speicherdauer-Konsistenz (DSI ↔ Cookie-Tabelle ↔ Realität)"
"</h2>"
"<p style='margin:0 0 8px;font-size:14px;color:#3f3f46;'>"
f"<strong>{total}</strong> Cookies verglichen: "
f"<strong style='color:#15803d;'>{passed} ✓</strong> / "
f"<strong style='color:#dc2626;'>{failed} ✗</strong> / "
f"<strong style='color:#64748b;'>{incomplete} ?</strong></p>"
"<table style='width:100%;border-collapse:collapse;font-size:13px;"
"margin-top:8px;background:#fff;'>"
"<thead><tr style='background:#f1f5f9;'>"
"<th style='text-align:left;padding:6px 10px;'>Cookie</th>"
"<th style='text-align:left;padding:6px 10px;'>Vendor</th>"
"<th style='text-align:left;padding:6px 10px;'>Werte</th>"
"<th style='text-align:left;padding:6px 10px;'>Mismatch</th>"
"</tr></thead>"
f"<tbody>{''.join(rows)}</tbody>"
"</table>"
"</div>"
)