feat(control-pipeline): incremental dedup + ENISA CRA ingestion
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-consent (push) Successful in 43s
CI / test-python-voice (push) Successful in 33s
CI / test-bqas (push) Successful in 37s
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-consent (push) Successful in 43s
CI / test-python-voice (push) Successful in 33s
CI / test-bqas (push) Successful in 37s
BatchDedup since-Parameter (services/batch_dedup_runner.py + api): - Neuer 'since: datetime' Param scoped Phase 1 + Phase 2 SQL auf created_at >= since. - Phase 2 checkpoint wird beim scoped Lauf geloescht (verhindert Skip neuer Atomics deren control_id alphabetisch unter dem stale last_id liegt). - 6-13x schneller fuer nachgeschobene Dokumente (19k statt 172k Atomics). - Doku: control-pipeline/docs/incremental-dedup.md. Neue Scripts: - gpre1_object_groups_incremental.py: Append neuer Objects an object_groups via bge-m3 nearest-neighbor (threshold default 0.85, empfehlbar 0.78 fuer breiteres Synonym-Matching). Pure INSERT/UPDATE, kein DELETE. - gpre2_master_controls_incremental.py: Non-destructive Master-Controls-Update. Existing MCs unangetastet (UUIDs + master_control_id bleiben), nur neue Members appended + neue MCs fuer Object-Groups die jetzt min-phases erreichen. - ingest_enisa_cra.py: Ingestion der 8 CRA-relevanten ENISA-Dokumente (Standards Mapping, EUCC-Implementation, NIS2 TIG, SRP FAQ, EUCC Eval Methodology, CVD Policies, Threat Landscape 2025). chunk_strategy=legal, requirement_strength=guidance|consultation_draft|evidentiary. Quelldaten: legal-sources/enisa/enisa_cra_single_reporting_platform_faq.html (PDFs sind .gitignore-gefiltert). Ergebnis dieser Pipeline-Iteration: - 1.296 neue CRA-Controls + 19.652 atomare Children - +362 neue Master-Controls, 10.017 existing erweitert - Total: 13.950 MCs, 620 CRA-MCs (vorher 566), 1.304 CRA-Atomics (vorher 841) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Benjamin Admin
@@ -1553,6 +1553,7 @@ async def get_repair_backfill_status(backfill_id: str):
|
||||
class BatchDedupRequest(BaseModel):
|
||||
dry_run: bool = True
|
||||
hint_filter: Optional[str] = None # Only process groups matching this hint prefix
|
||||
since: Optional[str] = None # ISO datetime — scope to controls created at/after this
|
||||
|
||||
|
||||
_batch_dedup_status: dict = {}
|
||||
@@ -1567,7 +1568,15 @@ async def _run_batch_dedup(req: BatchDedupRequest, dedup_id: str):
|
||||
runner = BatchDedupRunner(db)
|
||||
_batch_dedup_status[dedup_id] = {"status": "running", "phase": "starting"}
|
||||
|
||||
stats = await runner.run(dry_run=req.dry_run, hint_filter=req.hint_filter)
|
||||
since_dt = None
|
||||
if req.since:
|
||||
from datetime import datetime
|
||||
since_dt = datetime.fromisoformat(req.since.replace("Z", "+00:00"))
|
||||
stats = await runner.run(
|
||||
dry_run=req.dry_run,
|
||||
hint_filter=req.hint_filter,
|
||||
since=since_dt,
|
||||
)
|
||||
|
||||
_batch_dedup_status[dedup_id] = {
|
||||
"status": "completed",
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
# Incremental BatchDedup für nachgeschobene Dokumente
|
||||
|
||||
Eingefuehrt am 2026-05-18. Pattern fuer alle zukuenftigen Einzeldokument-Ingestionen.
|
||||
|
||||
## Problem
|
||||
|
||||
Der Default-BatchDedup-Runner lief gegen ALLE `pass0b` Atomics ohne Filter
|
||||
(WHERE decomposition_method = 'pass0b' AND release_state NOT IN ('deprecated','duplicate')).
|
||||
Das sind bei uns ~172k Controls. Pace ~5k/h → 25-40h Laufzeit. Bei jedem
|
||||
hinzugefuegten Dokument der gleiche volle Lauf — auch wenn das neue Dokument
|
||||
nur 1-2k Atomics erzeugt.
|
||||
|
||||
Zusaetzliches Risiko: Phase 1 schreibt master_controls erst am Ende. Ein
|
||||
Container-Crash mitten im Lauf (z.B. via Qdrant-Timeout) verwirft 100%
|
||||
des In-Memory-Fortschritts.
|
||||
|
||||
## Loesung — `since` Parameter
|
||||
|
||||
`POST /v1/canonical/generate/batch-dedup` akzeptiert jetzt:
|
||||
|
||||
```json
|
||||
{
|
||||
"dry_run": false,
|
||||
"since": "2026-05-18T02:53:00+00:00"
|
||||
}
|
||||
```
|
||||
|
||||
Effekt:
|
||||
- Phase 1 (intra-group dedup) laedt nur Controls mit `created_at >= since`
|
||||
- Phase 2 (cross-group dedup) filtert ebenfalls auf `created_at >= since`
|
||||
- Phase 2 Checkpoint wird vor Lauf-Start geloescht (sonst skippt stale
|
||||
`last_control_id` neu erzeugte Atomics deren control_id alphabetisch
|
||||
davor liegt)
|
||||
|
||||
Phase 2 sucht weiter im **vollen** Qdrant-Index `atomic_controls_dedup`,
|
||||
findet also Matches zu alten Master Controls und verlinkt korrekt.
|
||||
|
||||
## Wann verwenden
|
||||
|
||||
| Szenario | Empfehlung |
|
||||
|---|---|
|
||||
| Einzelnes neues Dokument ingestiert + Pass 0a + Pass 0b durchgelaufen | `since` setzen auf Zeitpunkt vor Pass 0b |
|
||||
| Mehrere kleine Updates seit letztem Full-Dedup | `since` setzen auf Zeitpunkt nach letztem Full-Dedup |
|
||||
| Initial-Setup oder Pipeline-Major-Update | KEIN `since` — full run |
|
||||
| Verdacht auf Drift / Quality-Regression | KEIN `since` — full run |
|
||||
|
||||
## Workflow nach Einzeldokument-Ingestion
|
||||
|
||||
```bash
|
||||
# 1. Pass 0a auf neue Controls (Obligations extrahieren)
|
||||
curl -X POST .../v1/canonical/generate/run-pass0a -d '{...}'
|
||||
|
||||
# 2. Pass 0b Decomposition Submit (Atomics erzeugen)
|
||||
curl -X POST .../v1/canonical/generate/submit-pass0b -d '{...}'
|
||||
|
||||
# 3. Wenn Anthropic Batch durch: process-batch
|
||||
curl -X POST .../v1/canonical/generate/process-batch -d '{
|
||||
"batch_id": "msgbatch_...",
|
||||
"pass_type": "0b"
|
||||
}'
|
||||
|
||||
# 4. Inkrementell deduppen (NEU, statt 25h full run)
|
||||
curl -X POST .../v1/canonical/generate/batch-dedup -d '{
|
||||
"dry_run": false,
|
||||
"since": "<ISO-Datetime kurz vor Pass-0b-Start>"
|
||||
}'
|
||||
```
|
||||
|
||||
## Pace-Beobachtung (CRA-Lauf 2026-05-18)
|
||||
|
||||
- Total neue Atomics: 19.423
|
||||
- Phase 1 multi-groups: 568 (Rest 18.101 sind Singletons → direkt Master)
|
||||
- Phase 2 Cross-Group: ~3-4h erwartet
|
||||
- Vergleich: Full-Run waere 25-40h gewesen, scoped 6-13x schneller.
|
||||
|
||||
## Implementation-Details (fuer Wartung)
|
||||
|
||||
Geaenderte Dateien:
|
||||
- `services/batch_dedup_runner.py` — `run()` + `_load_merge_groups()` +
|
||||
`_run_cross_group_pass()` SQL-Queries
|
||||
- `api/control_generator_routes.py` — `BatchDedupRequest.since` Feld +
|
||||
Handler reicht durch
|
||||
|
||||
Backwards-kompatibel: ohne `since` aequivalent zum alten Verhalten.
|
||||
|
||||
## Bekannte Limits
|
||||
|
||||
1. **Phase 2 Checkpoint wird beim scoped Lauf geloescht.** Wenn waehrend
|
||||
eines `since`-Laufs ein voller Run dazwischen geschoben werden soll
|
||||
(sollte nicht passieren), muss neu starten.
|
||||
2. **Phase 1 commit-Granularitaet nicht angefasst.** Bei Crash mitten in
|
||||
Phase 1 ohne `since` bleibt der Verlust gleich. Aber: scoped Phase 1
|
||||
ist so kurz (Minuten), dass das praktisch egal ist.
|
||||
3. **Singleton-Atomics werden direkt Master ohne Cross-Check.** Wenn ein
|
||||
neues Singleton-Atomic semantisch identisch zu einem alten Master
|
||||
ist, faengt das nur Phase 2 (via Qdrant). Funktioniert solange Phase 2
|
||||
nicht uebersprungen wird (dry_run=false ist Pflicht).
|
||||
|
||||
## Memory-Eintrag
|
||||
|
||||
Siehe `~/.claude/projects/-Users-benjaminadmin-Projekte-breakpilot-core/memory/feedback_incremental_dedup.md`
|
||||
@@ -0,0 +1,203 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
G-pre1 INCREMENTAL: Append new objects to object_groups via embedding similarity.
|
||||
|
||||
Non-destructive alternative to gpre1_object_clustering.py (which DELETEs and
|
||||
rebuilds all groups via K-Means). This script:
|
||||
- Finds objects referenced in atomic controls that are NOT yet in
|
||||
object_groups.members
|
||||
- Embeds each unmatched object via bge-m3 (local embedding-service)
|
||||
- Nearest-neighbor search against existing object_groups.canonical_name
|
||||
- Cosine >= --threshold (default 0.85) → APPEND to existing group's members
|
||||
- Cosine < --threshold → CREATE new object_group with next free group_id
|
||||
|
||||
Existing groups stay; only members get appended and new groups get added.
|
||||
|
||||
Usage (inside control-pipeline container):
|
||||
python3 /app/scripts/gpre1_object_groups_incremental.py --since 2026-05-18T02:53:00+00:00 --dry-run
|
||||
python3 /app/scripts/gpre1_object_groups_incremental.py --since 2026-05-18T02:53:00+00:00
|
||||
python3 /app/scripts/gpre1_object_groups_incremental.py --since 2026-05-18T02:53:00+00:00 --threshold 0.82
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime
|
||||
|
||||
import httpx
|
||||
import numpy as np
|
||||
from sqlalchemy import create_engine, text
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
|
||||
logger = logging.getLogger("gpre1_inc")
|
||||
|
||||
DB_URL = os.getenv("DATABASE_URL", "postgresql://breakpilot:breakpilot123@postgres:5432/breakpilot_db")
|
||||
EMBEDDING_URL = os.getenv("EMBEDDING_URL", "http://embedding-service:8087")
|
||||
BATCH_SIZE = 64
|
||||
|
||||
|
||||
def embed_batch(texts: list[str]) -> np.ndarray:
|
||||
"""Embed a list of strings via bge-m3 embedding-service."""
|
||||
with httpx.Client(timeout=120.0) as c:
|
||||
resp = c.post(f"{EMBEDDING_URL}/embed", json={"texts": texts, "normalize": True})
|
||||
resp.raise_for_status()
|
||||
return np.array(resp.json()["embeddings"], dtype=np.float32)
|
||||
|
||||
|
||||
def embed_many(texts: list[str], label: str = "") -> np.ndarray:
|
||||
"""Embed many strings in batches."""
|
||||
n = len(texts)
|
||||
out = np.zeros((n, 1024), dtype=np.float32)
|
||||
for i in range(0, n, BATCH_SIZE):
|
||||
batch = texts[i:i + BATCH_SIZE]
|
||||
out[i:i + len(batch)] = embed_batch(batch)
|
||||
if (i // BATCH_SIZE) % 20 == 0:
|
||||
logger.info(" %s: %d/%d embedded", label, i + len(batch), n)
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--since", required=True, help="ISO datetime — consider atomics from this date onwards")
|
||||
parser.add_argument("--threshold", type=float, default=0.85,
|
||||
help="Cosine threshold for appending to existing group (default 0.85)")
|
||||
parser.add_argument("--dry-run", action="store_true")
|
||||
args = parser.parse_args()
|
||||
|
||||
since_dt = datetime.fromisoformat(args.since.replace("Z", "+00:00"))
|
||||
logger.info("Incremental object_groups update since %s, threshold=%.2f, dry_run=%s",
|
||||
since_dt.isoformat(), args.threshold, args.dry_run)
|
||||
|
||||
engine = create_engine(DB_URL, connect_args={"options": "-c search_path=compliance,public"})
|
||||
|
||||
# 1. Load existing object_groups (id, canonical_name, members)
|
||||
with engine.connect() as c:
|
||||
rows = c.execute(text("""
|
||||
SELECT group_id, canonical_name, members FROM object_groups
|
||||
""")).fetchall()
|
||||
existing_groups = [(r[0], r[1], json.loads(r[2]) if isinstance(r[2], str) else r[2]) for r in rows]
|
||||
logger.info("Loaded %d existing object_groups", len(existing_groups))
|
||||
|
||||
existing_members: set[str] = set()
|
||||
for _, _, members in existing_groups:
|
||||
for m in members:
|
||||
existing_members.add(m)
|
||||
logger.info("Existing union of members: %d distinct strings", len(existing_members))
|
||||
|
||||
# 2. Find unmatched objects from atomics since `since`
|
||||
from services.control_dedup import normalize_object
|
||||
with engine.connect() as c:
|
||||
rows = c.execute(text("""
|
||||
SELECT DISTINCT split_part(generation_metadata->>'merge_group_hint', ':', 2) AS obj
|
||||
FROM canonical_controls
|
||||
WHERE decomposition_method = 'pass0b'
|
||||
AND created_at >= :since
|
||||
AND generation_metadata->>'merge_group_hint' IS NOT NULL
|
||||
AND generation_metadata->>'merge_group_hint' != ''
|
||||
AND release_state NOT IN ('deprecated', 'rejected', 'duplicate')
|
||||
"""), {"since": since_dt}).fetchall()
|
||||
new_objects_raw = [r[0] for r in rows if r[0]]
|
||||
logger.info("Distinct objects in new atomics: %d", len(new_objects_raw))
|
||||
|
||||
# Normalize each + dedupe; track originals → normalized
|
||||
normed_to_originals: dict[str, set[str]] = {}
|
||||
for obj in new_objects_raw:
|
||||
normed = normalize_object(obj)
|
||||
if not normed:
|
||||
continue
|
||||
if normed in existing_members or obj in existing_members:
|
||||
continue # already in some group
|
||||
normed_to_originals.setdefault(normed, set()).update([normed, obj])
|
||||
|
||||
unmatched_normed = list(normed_to_originals.keys())
|
||||
logger.info("Unmatched normalized objects: %d", len(unmatched_normed))
|
||||
|
||||
if not unmatched_normed:
|
||||
logger.info("Nothing to do — all objects already mapped.")
|
||||
return
|
||||
|
||||
# 3. Embed existing canonical_names + unmatched objects
|
||||
logger.info("Embedding %d existing canonical_names...", len(existing_groups))
|
||||
existing_emb = embed_many([g[1] for g in existing_groups], label="existing")
|
||||
logger.info("Embedding %d unmatched objects...", len(unmatched_normed))
|
||||
unmatched_emb = embed_many(unmatched_normed, label="unmatched")
|
||||
|
||||
# 4. Nearest-neighbor: for each unmatched, find best existing match
|
||||
# cosine = dot product (both already L2-normalized)
|
||||
logger.info("Computing nearest-neighbor matches...")
|
||||
sims = unmatched_emb @ existing_emb.T # (N_unmatched, N_existing)
|
||||
best_idx = sims.argmax(axis=1)
|
||||
best_score = sims.max(axis=1)
|
||||
|
||||
appends: dict[int, list[str]] = {} # group_id → list of new members
|
||||
new_groups: list[tuple[str, list[str]]] = [] # (canonical_name, members)
|
||||
|
||||
for i, normed in enumerate(unmatched_normed):
|
||||
originals = sorted(normed_to_originals[normed])
|
||||
if best_score[i] >= args.threshold:
|
||||
gid = existing_groups[int(best_idx[i])][0]
|
||||
appends.setdefault(gid, []).extend(originals)
|
||||
else:
|
||||
# Create a new group with this object as canonical
|
||||
new_groups.append((normed, originals))
|
||||
|
||||
# Stats
|
||||
distinct_groups_to_extend = len(appends)
|
||||
total_appends = sum(len(v) for v in appends.values())
|
||||
logger.info("Plan: extend %d existing groups (+%d members), create %d new groups",
|
||||
distinct_groups_to_extend, total_appends, len(new_groups))
|
||||
|
||||
if args.dry_run:
|
||||
logger.info("DRY RUN — no writes")
|
||||
# Sample
|
||||
if appends:
|
||||
sample = list(appends.items())[:5]
|
||||
for gid, members in sample:
|
||||
gname = next((g[1] for g in existing_groups if g[0] == gid), "?")
|
||||
logger.info(" Extend group_id=%d (%s) with: %s", gid, gname, members[:3])
|
||||
if new_groups:
|
||||
for name, members in new_groups[:5]:
|
||||
logger.info(" NEW group: %s — members=%s", name, members[:3])
|
||||
return
|
||||
|
||||
# 5. Write — pure INSERT/UPDATE
|
||||
with engine.begin() as c:
|
||||
c.execute(text("SET search_path TO compliance, public"))
|
||||
|
||||
# UPDATE existing groups (append to members JSONB)
|
||||
for gid, new_members in appends.items():
|
||||
c.execute(text("""
|
||||
UPDATE object_groups
|
||||
SET members = (
|
||||
SELECT jsonb_agg(DISTINCT m)
|
||||
FROM jsonb_array_elements_text(members || CAST(:new_members AS jsonb)) AS x(m)
|
||||
),
|
||||
member_count = (
|
||||
SELECT count(DISTINCT m)
|
||||
FROM jsonb_array_elements_text(members || CAST(:new_members AS jsonb)) AS x(m)
|
||||
)
|
||||
WHERE group_id = :gid
|
||||
"""), {"gid": gid, "new_members": json.dumps(new_members)})
|
||||
|
||||
# INSERT new groups with next free group_id
|
||||
next_gid_row = c.execute(text("SELECT COALESCE(MAX(group_id), 0) + 1 FROM object_groups")).fetchone()
|
||||
next_gid = next_gid_row[0] if next_gid_row else 1
|
||||
for name, members in new_groups:
|
||||
c.execute(text("""
|
||||
INSERT INTO object_groups (group_id, canonical_name, member_count, members, top_controls_count)
|
||||
VALUES (:gid, :name, :count, CAST(:members AS jsonb), 0)
|
||||
"""), {
|
||||
"gid": next_gid,
|
||||
"name": name[:200],
|
||||
"count": len(members),
|
||||
"members": json.dumps(members),
|
||||
})
|
||||
next_gid += 1
|
||||
|
||||
logger.info("DONE — extended %d existing groups (+%d members), created %d new groups",
|
||||
distinct_groups_to_extend, total_appends, len(new_groups))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,267 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
G-pre2 INCREMENTAL: Add new atomic controls to Master Controls without rebuild.
|
||||
|
||||
Unlike gpre2_master_controls.py which DELETEs and rebuilds the entire
|
||||
master_controls table, this script is non-destructive:
|
||||
- Existing master_controls stay untouched (same UUIDs, same MC-IDs)
|
||||
- For each object_group that gained new atomic controls:
|
||||
* If MC exists: append new members + update total_controls/phase_counts
|
||||
* If MC missing AND group now has >= min_phases: create new MC + all members
|
||||
|
||||
Usage:
|
||||
python3 /app/scripts/gpre2_master_controls_incremental.py --since 2026-05-18T02:53:00+00:00
|
||||
python3 /app/scripts/gpre2_master_controls_incremental.py --since 2026-05-18T02:53:00+00:00 --dry-run
|
||||
python3 /app/scripts/gpre2_master_controls_incremental.py --since 2026-05-18T02:53:00+00:00 --min-phases 2
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from collections import defaultdict
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import create_engine, text
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
|
||||
logger = logging.getLogger("gpre2_incremental")
|
||||
|
||||
DB_URL = os.getenv("DATABASE_URL", "postgresql://breakpilot:breakpilot123@postgres:5432/breakpilot_db")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--since", required=True, help="ISO datetime — only consider atomics created at/after this")
|
||||
parser.add_argument("--min-phases", type=int, default=2, help="Min distinct phases to form a new MC (default 2)")
|
||||
parser.add_argument("--dry-run", action="store_true")
|
||||
args = parser.parse_args()
|
||||
|
||||
since_dt = datetime.fromisoformat(args.since.replace("Z", "+00:00"))
|
||||
logger.info("Incremental run since %s, min_phases=%d, dry_run=%s",
|
||||
since_dt.isoformat(), args.min_phases, args.dry_run)
|
||||
|
||||
engine = create_engine(DB_URL, connect_args={"options": "-c search_path=compliance,public"})
|
||||
|
||||
# Step 1: object → group_id reverse index
|
||||
object_to_group = {}
|
||||
with engine.connect() as c:
|
||||
groups = c.execute(text("SELECT group_id, canonical_name, members FROM object_groups")).fetchall()
|
||||
for gid, canonical, members_json in groups:
|
||||
members = json.loads(members_json) if isinstance(members_json, str) else members_json
|
||||
for member in members:
|
||||
object_to_group[member] = (gid, canonical)
|
||||
logger.info("Reverse index: %d objects → %d groups", len(object_to_group), len(groups))
|
||||
|
||||
# Step 2: Load ALL atomics with merge_group_hint (we need full picture)
|
||||
with engine.connect() as c:
|
||||
all_rows = c.execute(text("""
|
||||
SELECT id, control_id,
|
||||
generation_metadata->>'merge_group_hint' AS hint,
|
||||
title,
|
||||
created_at
|
||||
FROM canonical_controls
|
||||
WHERE generation_metadata->>'merge_group_hint' IS NOT NULL
|
||||
AND generation_metadata->>'merge_group_hint' != ''
|
||||
AND release_state NOT IN ('deprecated', 'rejected', 'duplicate')
|
||||
""")).fetchall()
|
||||
logger.info("Loaded %d atomic controls total", len(all_rows))
|
||||
|
||||
# Step 3: Build group_phases (gid → phase → [(uuid, control_id, action, title, is_new)])
|
||||
from services.control_dedup import normalize_object
|
||||
group_phases: dict[int, dict[str, list]] = defaultdict(lambda: defaultdict(list))
|
||||
group_names: dict[int, str] = {}
|
||||
new_atomic_count = 0
|
||||
new_groups_touched: set[int] = set()
|
||||
unmatched = 0
|
||||
|
||||
for uuid, control_id, hint, title, created_at in all_rows:
|
||||
parts = hint.split(":", 2)
|
||||
if len(parts) < 2:
|
||||
continue
|
||||
action = parts[0]
|
||||
obj = parts[1]
|
||||
phase = parts[2] if len(parts) > 2 else "implementation"
|
||||
normed = normalize_object(obj)
|
||||
if normed in object_to_group:
|
||||
gid, canonical = object_to_group[normed]
|
||||
elif obj in object_to_group:
|
||||
gid, canonical = object_to_group[obj]
|
||||
else:
|
||||
unmatched += 1
|
||||
continue
|
||||
is_new = created_at >= since_dt
|
||||
group_phases[gid][phase].append((str(uuid), control_id, action, title, is_new))
|
||||
group_names[gid] = canonical
|
||||
if is_new:
|
||||
new_atomic_count += 1
|
||||
new_groups_touched.add(gid)
|
||||
|
||||
logger.info("Total: %d new atomics across %d object_groups (%d unmatched)",
|
||||
new_atomic_count, len(new_groups_touched), unmatched)
|
||||
|
||||
if not new_groups_touched:
|
||||
logger.info("Nothing to do — no new atomics matched to any object_group.")
|
||||
return
|
||||
|
||||
# Step 4: For each touched object_group, decide action
|
||||
stats = {
|
||||
"groups_examined": len(new_groups_touched),
|
||||
"mcs_existing_updated": 0,
|
||||
"mcs_new_created": 0,
|
||||
"members_inserted": 0,
|
||||
"members_skipped_existing": 0,
|
||||
"groups_skipped_below_min_phases": 0,
|
||||
"groups_skipped_no_member_change": 0,
|
||||
}
|
||||
|
||||
# Load existing master_controls index: master_control_id → uuid
|
||||
with engine.connect() as c:
|
||||
mc_index = {row[1]: (str(row[0]), row[2]) for row in c.execute(text(
|
||||
"SELECT id, master_control_id, total_controls FROM master_controls"
|
||||
)).fetchall()}
|
||||
logger.info("Existing master_controls: %d", len(mc_index))
|
||||
|
||||
# Load existing members for touched MCs (avoid duplicate inserts)
|
||||
touched_mc_ids = ["MC-%d" % gid for gid in new_groups_touched]
|
||||
existing_members: dict[str, set[str]] = defaultdict(set)
|
||||
with engine.connect() as c:
|
||||
for mc_id_str in touched_mc_ids:
|
||||
mc_uuid_info = mc_index.get(mc_id_str)
|
||||
if not mc_uuid_info:
|
||||
continue
|
||||
mc_uuid = mc_uuid_info[0]
|
||||
for row in c.execute(text(
|
||||
"SELECT control_uuid FROM master_control_members WHERE master_control_uuid = CAST(:u AS uuid)"
|
||||
), {"u": mc_uuid}).fetchall():
|
||||
existing_members[mc_id_str].add(str(row[0]))
|
||||
|
||||
# Build INSERT/UPDATE plans
|
||||
inserts_new_mcs = []
|
||||
inserts_members = []
|
||||
updates_mcs = []
|
||||
|
||||
PHASE_ORDER = {
|
||||
"scope": 0, "definition": 1, "governance": 1, "design": 2,
|
||||
"implementation": 3, "configuration": 3, "operation": 4, "training": 4,
|
||||
"monitoring": 5, "testing": 6, "review": 7, "assessment": 8,
|
||||
"remediation": 8, "validation": 9, "reporting": 10, "evidence": 11,
|
||||
}
|
||||
|
||||
for gid in new_groups_touched:
|
||||
mc_id_str = "MC-%d" % gid
|
||||
phases = group_phases[gid]
|
||||
canonical = group_names[gid]
|
||||
all_phases = sorted(phases.keys(), key=lambda p: PHASE_ORDER.get(p, 99))
|
||||
phase_counts = {p: len(ctrls) for p, ctrls in phases.items()}
|
||||
total = sum(phase_counts.values())
|
||||
|
||||
existing_mc = mc_index.get(mc_id_str)
|
||||
|
||||
if existing_mc:
|
||||
# MC exists — append only NEW atomics that aren't already members
|
||||
mc_uuid = existing_mc[0]
|
||||
existing_set = existing_members[mc_id_str]
|
||||
added_for_this_mc = 0
|
||||
for phase, controls in phases.items():
|
||||
for ctrl_uuid, ctrl_id, action, title, is_new in controls:
|
||||
if ctrl_uuid in existing_set:
|
||||
stats["members_skipped_existing"] += 1
|
||||
continue
|
||||
inserts_members.append({
|
||||
"mc_uuid": mc_uuid, "control_uuid": ctrl_uuid,
|
||||
"phase": phase, "action": action,
|
||||
})
|
||||
stats["members_inserted"] += 1
|
||||
added_for_this_mc += 1
|
||||
if added_for_this_mc > 0:
|
||||
updates_mcs.append({
|
||||
"mc_uuid": mc_uuid,
|
||||
"phases_covered": json.dumps(all_phases),
|
||||
"phase_control_count": json.dumps(phase_counts),
|
||||
"total_controls": total,
|
||||
})
|
||||
stats["mcs_existing_updated"] += 1
|
||||
else:
|
||||
stats["groups_skipped_no_member_change"] += 1
|
||||
else:
|
||||
# MC missing — create only if group now meets min_phases threshold
|
||||
if len(phases) < args.min_phases:
|
||||
stats["groups_skipped_below_min_phases"] += 1
|
||||
continue
|
||||
inserts_new_mcs.append({
|
||||
"master_control_id": mc_id_str,
|
||||
"object_group_id": gid,
|
||||
"canonical_name": canonical,
|
||||
"phases_covered": json.dumps(all_phases),
|
||||
"phase_control_count": json.dumps(phase_counts),
|
||||
"total_controls": total,
|
||||
"_members": [
|
||||
{"control_uuid": c[0], "phase": p, "action": c[2]}
|
||||
for p, ctrls in phases.items() for c in ctrls
|
||||
],
|
||||
})
|
||||
stats["mcs_new_created"] += 1
|
||||
|
||||
logger.info("Plan summary: %s", stats)
|
||||
|
||||
if args.dry_run:
|
||||
logger.info("DRY RUN — no writes")
|
||||
# Show first few examples
|
||||
if inserts_new_mcs:
|
||||
logger.info("Sample NEW MCs (up to 5):")
|
||||
for mc in inserts_new_mcs[:5]:
|
||||
logger.info(" %s: %s — total=%d, phases=%s",
|
||||
mc["master_control_id"], mc["canonical_name"],
|
||||
mc["total_controls"], mc["phases_covered"])
|
||||
if updates_mcs:
|
||||
logger.info("Updates to existing MCs: %d", len(updates_mcs))
|
||||
return
|
||||
|
||||
# Step 5: WRITE — strictly INSERT/UPDATE, no DELETE
|
||||
with engine.begin() as c:
|
||||
c.execute(text("SET search_path TO compliance, public"))
|
||||
|
||||
# 5a: Insert new MCs + their members
|
||||
for mc in inserts_new_mcs:
|
||||
new_uuid_row = c.execute(text("""
|
||||
INSERT INTO master_controls
|
||||
(master_control_id, object_group_id, canonical_name,
|
||||
phases_covered, phase_control_count, total_controls)
|
||||
VALUES (:master_control_id, :object_group_id, :canonical_name,
|
||||
CAST(:phases_covered AS jsonb), CAST(:phase_control_count AS jsonb),
|
||||
:total_controls)
|
||||
RETURNING id
|
||||
"""), {k: v for k, v in mc.items() if k != "_members"}).fetchone()
|
||||
new_mc_uuid = str(new_uuid_row[0])
|
||||
for mem in mc["_members"]:
|
||||
c.execute(text("""
|
||||
INSERT INTO master_control_members
|
||||
(master_control_uuid, control_uuid, phase, action)
|
||||
VALUES (CAST(:mc_uuid AS uuid), CAST(:control_uuid AS uuid), :phase, :action)
|
||||
"""), {"mc_uuid": new_mc_uuid, **mem})
|
||||
|
||||
# 5b: Append new members to existing MCs
|
||||
for mem in inserts_members:
|
||||
c.execute(text("""
|
||||
INSERT INTO master_control_members
|
||||
(master_control_uuid, control_uuid, phase, action)
|
||||
VALUES (CAST(:mc_uuid AS uuid), CAST(:control_uuid AS uuid), :phase, :action)
|
||||
"""), mem)
|
||||
|
||||
# 5c: Update phase counts / totals on touched existing MCs
|
||||
for upd in updates_mcs:
|
||||
c.execute(text("""
|
||||
UPDATE master_controls
|
||||
SET phases_covered = CAST(:phases_covered AS jsonb),
|
||||
phase_control_count = CAST(:phase_control_count AS jsonb),
|
||||
total_controls = :total_controls
|
||||
WHERE id = CAST(:mc_uuid AS uuid)
|
||||
"""), upd)
|
||||
|
||||
logger.info("DONE — wrote %d new MCs, updated %d existing MCs, %d members inserted",
|
||||
stats["mcs_new_created"], stats["mcs_existing_updated"], stats["members_inserted"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,414 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Ingest CRA-relevant ENISA documents into the RAG (collection `bp_compliance_ce`).
|
||||
|
||||
Source files live under `legal-sources/enisa/` in this repo. The script extracts
|
||||
PDF text with pdfplumber (HTML for the SRP FAQ), normalizes it, and uploads via
|
||||
the RAG service with `chunk_strategy='legal'` so that section metadata is
|
||||
attached to every chunk.
|
||||
|
||||
Each document carries a `requirement_strength` field so downstream consumers
|
||||
can distinguish normative material from guidance and consultation drafts:
|
||||
- mandatory — binding (none in this batch; CRA itself is the law)
|
||||
- guidance — official ENISA / EUCC guidance, citable
|
||||
- consultation_draft — public-consultation drafts (use with caveat)
|
||||
|
||||
Usage (run on Mac Mini after copying the legal-sources/enisa/ folder, or via SSH
|
||||
with the repo mounted):
|
||||
python3 control-pipeline/scripts/ingest_enisa_cra.py --dry-run
|
||||
python3 control-pipeline/scripts/ingest_enisa_cra.py
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
import unicodedata
|
||||
from html.parser import HTMLParser
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pdfplumber
|
||||
|
||||
RAG_URL = "https://localhost:8097"
|
||||
QDRANT_URL = "http://localhost:6333"
|
||||
UPLOAD_TIMEOUT = 1800.0
|
||||
COLLECTION = "bp_compliance_ce"
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
SOURCE_DIR = REPO_ROOT / "legal-sources" / "enisa"
|
||||
|
||||
DOCS = [
|
||||
{
|
||||
"regulation_id": "enisa_cra_requirements_standards_mapping",
|
||||
"filename": "enisa_cra_requirements_standards_mapping.pdf",
|
||||
"upload_filename": "enisa_cra_requirements_standards_mapping.txt",
|
||||
"extra_metadata": {
|
||||
"regulation_id": "enisa_cra_requirements_standards_mapping",
|
||||
"regulation_short": "ENISA CRA Standards Mapping",
|
||||
"guideline_name": "Cyber Resilience Act Requirements Standards Mapping",
|
||||
"doc_type": "standards_mapping",
|
||||
"requirement_strength": "guidance",
|
||||
"publication_year": "2024",
|
||||
"license": "reuse_with_attribution",
|
||||
"source": "enisa.europa.eu",
|
||||
"attribution": "ENISA, CC BY 4.0",
|
||||
},
|
||||
},
|
||||
{
|
||||
"regulation_id": "enisa_cra_implementation_via_eucc",
|
||||
"filename": "enisa_cra_implementation_via_eucc.pdf",
|
||||
"upload_filename": "enisa_cra_implementation_via_eucc.txt",
|
||||
"extra_metadata": {
|
||||
"regulation_id": "enisa_cra_implementation_via_eucc",
|
||||
"regulation_short": "ENISA CRA via EUCC",
|
||||
"guideline_name": "CRA Implementation via EUCC and its Applicable Technical Elements",
|
||||
"doc_type": "certification_guidance",
|
||||
"requirement_strength": "guidance",
|
||||
"license": "reuse_with_attribution",
|
||||
"source": "enisa.europa.eu",
|
||||
"attribution": "ENISA, CC BY 4.0",
|
||||
},
|
||||
},
|
||||
{
|
||||
"regulation_id": "enisa_cra_implementation_via_eucc_annex",
|
||||
"filename": "enisa_cra_implementation_via_eucc_annex.pdf",
|
||||
"upload_filename": "enisa_cra_implementation_via_eucc_annex.txt",
|
||||
"extra_metadata": {
|
||||
"regulation_id": "enisa_cra_implementation_via_eucc_annex",
|
||||
"regulation_short": "ENISA CRA via EUCC (Annex)",
|
||||
"guideline_name": "Annex — CRA Implementation via EUCC",
|
||||
"doc_type": "certification_guidance_annex",
|
||||
"requirement_strength": "guidance",
|
||||
"license": "reuse_with_attribution",
|
||||
"source": "enisa.europa.eu",
|
||||
"attribution": "ENISA, CC BY 4.0",
|
||||
},
|
||||
},
|
||||
{
|
||||
"regulation_id": "enisa_eucc_vulnerability_management_disclosure",
|
||||
"filename": "enisa_eucc_vulnerability_management_disclosure.pdf",
|
||||
"upload_filename": "enisa_eucc_vulnerability_management_disclosure.txt",
|
||||
"extra_metadata": {
|
||||
"regulation_id": "enisa_eucc_vulnerability_management_disclosure",
|
||||
"regulation_short": "EUCC Vuln Management & Disclosure",
|
||||
"guideline_name": "EUCC Guidelines — Vulnerability Management and Disclosure v1.1",
|
||||
"doc_type": "vulnerability_guidance",
|
||||
"requirement_strength": "guidance",
|
||||
"license": "reuse_with_attribution",
|
||||
"source": "enisa.europa.eu",
|
||||
"attribution": "ENISA, CC BY 4.0",
|
||||
},
|
||||
},
|
||||
{
|
||||
"regulation_id": "enisa_eccg_opinion_vulnerability_management",
|
||||
"filename": "enisa_eccg_opinion_vulnerability_management.pdf",
|
||||
"upload_filename": "enisa_eccg_opinion_vulnerability_management.txt",
|
||||
"extra_metadata": {
|
||||
"regulation_id": "enisa_eccg_opinion_vulnerability_management",
|
||||
"regulation_short": "ECCG Opinion Vuln Management",
|
||||
"guideline_name": "Final ECCG Opinion — Guidance on Vulnerability Management",
|
||||
"doc_type": "eccg_opinion",
|
||||
"requirement_strength": "guidance",
|
||||
"license": "reuse_with_attribution",
|
||||
"source": "enisa.europa.eu",
|
||||
"attribution": "ENISA, CC BY 4.0",
|
||||
},
|
||||
},
|
||||
{
|
||||
"regulation_id": "enisa_nis2_technical_implementation_guidance",
|
||||
"filename": "enisa_nis2_technical_implementation_guidance.pdf",
|
||||
"upload_filename": "enisa_nis2_technical_implementation_guidance.txt",
|
||||
"extra_metadata": {
|
||||
"regulation_id": "enisa_nis2_technical_implementation_guidance",
|
||||
"regulation_short": "ENISA NIS2 TIG v1.0",
|
||||
"guideline_name": "ENISA Technical Implementation Guidance on Cybersecurity Risk Management Measures v1.0",
|
||||
"doc_type": "technical_guidance",
|
||||
"requirement_strength": "guidance",
|
||||
"publication_year": "2025",
|
||||
"license": "reuse_with_attribution",
|
||||
"source": "enisa.europa.eu",
|
||||
"attribution": "ENISA, CC BY 4.0",
|
||||
},
|
||||
},
|
||||
{
|
||||
"regulation_id": "enisa_nis2_security_measures_consultation",
|
||||
"filename": "enisa_nis2_security_measures_implementation_guidance_consultation.pdf",
|
||||
"upload_filename": "enisa_nis2_security_measures_consultation.txt",
|
||||
"extra_metadata": {
|
||||
"regulation_id": "enisa_nis2_security_measures_consultation",
|
||||
"regulation_short": "ENISA NIS2 Security Measures (Draft)",
|
||||
"guideline_name": "Implementation Guidance on Security Measures — Public Consultation Draft",
|
||||
"doc_type": "consultation_draft",
|
||||
"requirement_strength": "consultation_draft",
|
||||
"license": "reuse_with_attribution",
|
||||
"source": "enisa.europa.eu",
|
||||
"attribution": "ENISA, CC BY 4.0",
|
||||
},
|
||||
},
|
||||
{
|
||||
"regulation_id": "enisa_cra_single_reporting_platform_faq",
|
||||
"filename": "enisa_cra_single_reporting_platform_faq.html",
|
||||
"upload_filename": "enisa_cra_single_reporting_platform_faq.txt",
|
||||
"extra_metadata": {
|
||||
"regulation_id": "enisa_cra_single_reporting_platform_faq",
|
||||
"regulation_short": "ENISA SRP FAQ",
|
||||
"guideline_name": "CRA Single Reporting Platform (SRP) FAQ",
|
||||
"doc_type": "faq",
|
||||
"requirement_strength": "guidance",
|
||||
"license": "reuse_with_attribution",
|
||||
"source": "enisa.europa.eu",
|
||||
"attribution": "ENISA, CC BY 4.0",
|
||||
},
|
||||
},
|
||||
{
|
||||
"regulation_id": "enisa_eucc_evaluation_methodology_product_series",
|
||||
"filename": "enisa_eucc_evaluation_methodology_product_series.pdf",
|
||||
"upload_filename": "enisa_eucc_evaluation_methodology_product_series.txt",
|
||||
"extra_metadata": {
|
||||
"regulation_id": "enisa_eucc_evaluation_methodology_product_series",
|
||||
"regulation_short": "EUCC Eval Methodology Product Series",
|
||||
"guideline_name": "EUCC Guidelines — Evaluation Methodology for Product Series v1.0",
|
||||
"doc_type": "evaluation_methodology",
|
||||
"requirement_strength": "guidance",
|
||||
"publication_year": "2025",
|
||||
"license": "reuse_with_attribution",
|
||||
"source": "enisa.europa.eu",
|
||||
"attribution": "ENISA, CC BY 4.0",
|
||||
},
|
||||
},
|
||||
{
|
||||
"regulation_id": "enisa_threat_landscape_2025",
|
||||
"filename": "enisa_threat_landscape_2025.pdf",
|
||||
"upload_filename": "enisa_threat_landscape_2025.txt",
|
||||
"extra_metadata": {
|
||||
"regulation_id": "enisa_threat_landscape_2025",
|
||||
"regulation_short": "ENISA Threat Landscape 2025",
|
||||
"guideline_name": "ENISA Threat Landscape 2025 v1.2",
|
||||
"doc_type": "threat_landscape",
|
||||
"requirement_strength": "evidentiary",
|
||||
"publication_year": "2025",
|
||||
"license": "reuse_with_attribution",
|
||||
"source": "enisa.europa.eu",
|
||||
"attribution": "ENISA, CC BY 4.0",
|
||||
},
|
||||
},
|
||||
{
|
||||
"regulation_id": "enisa_cvd_policies_eu_2022",
|
||||
"filename": "enisa_cvd_policies_eu_2022.pdf",
|
||||
"upload_filename": "enisa_cvd_policies_eu_2022.txt",
|
||||
"extra_metadata": {
|
||||
"regulation_id": "enisa_cvd_policies_eu_2022",
|
||||
"regulation_short": "ENISA CVD Policies EU 2022",
|
||||
"guideline_name": "Coordinated Vulnerability Disclosure Policies in the EU (2022)",
|
||||
"doc_type": "policy_study",
|
||||
"requirement_strength": "guidance",
|
||||
"publication_year": "2022",
|
||||
"license": "reuse_with_attribution",
|
||||
"source": "enisa.europa.eu",
|
||||
"attribution": "ENISA, CC BY 4.0",
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def normalize_text(text: str) -> str:
|
||||
text = unicodedata.normalize("NFKC", text)
|
||||
text = text.replace("", "").replace("", "")
|
||||
prev = None
|
||||
while prev != text:
|
||||
prev = text
|
||||
text = re.sub(r"(\d+)\s+\.\s+(\d+)", r"\1.\2", text)
|
||||
text = re.sub(r"\b([A-Z]{2,4})\s+-\s+(\d+)\b", r"\1-\2", text)
|
||||
text = re.sub(r"\(\s+(\d+)\s+\)", r"(\1)", text)
|
||||
text = re.sub(r"[^\S\n]{2,}", " ", text)
|
||||
return text
|
||||
|
||||
|
||||
class _HTMLToText(HTMLParser):
|
||||
SKIP = {"script", "style", "nav", "header", "footer", "noscript"}
|
||||
BLOCK = {"p", "div", "li", "br", "h1", "h2", "h3", "h4", "h5", "h6", "tr", "section"}
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self._buf: list[str] = []
|
||||
self._skip_depth = 0
|
||||
|
||||
def handle_starttag(self, tag, attrs):
|
||||
if tag in self.SKIP:
|
||||
self._skip_depth += 1
|
||||
if tag in self.BLOCK:
|
||||
self._buf.append("\n")
|
||||
|
||||
def handle_endtag(self, tag):
|
||||
if tag in self.SKIP and self._skip_depth > 0:
|
||||
self._skip_depth -= 1
|
||||
if tag in self.BLOCK:
|
||||
self._buf.append("\n")
|
||||
|
||||
def handle_data(self, data):
|
||||
if self._skip_depth == 0:
|
||||
self._buf.append(data)
|
||||
|
||||
def text(self) -> str:
|
||||
raw = "".join(self._buf)
|
||||
raw = re.sub(r"\n{3,}", "\n\n", raw)
|
||||
return raw.strip()
|
||||
|
||||
|
||||
def extract_pdf(path: Path) -> str:
|
||||
print(f" Extracting PDF: {path.name}")
|
||||
parts: list[str] = []
|
||||
with pdfplumber.open(path) as pdf:
|
||||
for i, page in enumerate(pdf.pages):
|
||||
t = page.extract_text(x_tolerance=3, y_tolerance=4)
|
||||
if t:
|
||||
parts.append(t)
|
||||
if (i + 1) % 50 == 0:
|
||||
print(f" {i + 1}/{len(pdf.pages)} pages...")
|
||||
return normalize_text("\n\n".join(parts))
|
||||
|
||||
|
||||
def extract_html(path: Path) -> str:
|
||||
print(f" Extracting HTML: {path.name}")
|
||||
html = path.read_text(encoding="utf-8", errors="replace")
|
||||
parser = _HTMLToText()
|
||||
parser.feed(html)
|
||||
return normalize_text(parser.text())
|
||||
|
||||
|
||||
def get_text(doc) -> str:
|
||||
path = SOURCE_DIR / doc["filename"]
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(path)
|
||||
if path.suffix.lower() == ".pdf":
|
||||
text = extract_pdf(path)
|
||||
elif path.suffix.lower() in {".html", ".htm"}:
|
||||
text = extract_html(path)
|
||||
else:
|
||||
raise ValueError(f"Unsupported file type: {path.suffix}")
|
||||
print(f" Extracted {len(text):,} chars")
|
||||
return text
|
||||
|
||||
|
||||
def upload_text_legal(text: str, filename: str, extra_metadata: dict) -> dict:
|
||||
form_data = {
|
||||
"collection": COLLECTION,
|
||||
"data_type": "compliance",
|
||||
"bundesland": "bund",
|
||||
"use_case": "compliance",
|
||||
"year": "2026",
|
||||
"chunk_strategy": "legal",
|
||||
"chunk_size": "1500",
|
||||
"chunk_overlap": "100",
|
||||
"metadata_json": json.dumps(extra_metadata, ensure_ascii=False),
|
||||
}
|
||||
with httpx.Client(timeout=UPLOAD_TIMEOUT, verify=False) as c:
|
||||
resp = c.post(
|
||||
f"{RAG_URL}/api/v1/documents/upload",
|
||||
files={"file": (filename, text.encode("utf-8"), "text/plain")},
|
||||
data=form_data,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
|
||||
def count_chunks(regulation_id: str) -> int:
|
||||
with httpx.Client(timeout=30) as c:
|
||||
resp = c.post(
|
||||
f"{QDRANT_URL}/collections/{COLLECTION}/points/count",
|
||||
json={
|
||||
"filter": {
|
||||
"must": [
|
||||
{"key": "regulation_id", "match": {"value": regulation_id}}
|
||||
]
|
||||
},
|
||||
"exact": True,
|
||||
},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()["result"]["count"]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--dry-run", action="store_true",
|
||||
help="Extract text and report sizes, but do not upload.")
|
||||
parser.add_argument("--only", action="append", default=[],
|
||||
help="Limit run to one or more regulation_ids.")
|
||||
args = parser.parse_args()
|
||||
|
||||
if not SOURCE_DIR.exists():
|
||||
print(f"ERROR: source dir not found: {SOURCE_DIR}")
|
||||
return 2
|
||||
|
||||
docs = DOCS
|
||||
if args.only:
|
||||
wanted = set(args.only)
|
||||
docs = [d for d in DOCS if d["regulation_id"] in wanted]
|
||||
missing = wanted - {d["regulation_id"] for d in docs}
|
||||
if missing:
|
||||
print(f"ERROR: unknown regulation_id(s): {sorted(missing)}")
|
||||
return 2
|
||||
|
||||
print("=" * 70)
|
||||
print(f"ENISA CRA ingestion → collection={COLLECTION}")
|
||||
print(f"Source dir: {SOURCE_DIR}")
|
||||
print(f"Documents: {len(docs)} Dry run: {args.dry_run}")
|
||||
print("=" * 70)
|
||||
|
||||
results = []
|
||||
for i, doc in enumerate(docs, 1):
|
||||
reg_id = doc["regulation_id"]
|
||||
print(f"\n[{i}/{len(docs)}] {reg_id}")
|
||||
|
||||
existing = count_chunks(reg_id) if not args.dry_run else "?"
|
||||
print(f" Existing chunks in Qdrant: {existing}")
|
||||
|
||||
try:
|
||||
text = get_text(doc)
|
||||
except Exception as e:
|
||||
print(f" ERROR extracting text: {e}")
|
||||
results.append({"id": reg_id, "chars": 0, "new": 0,
|
||||
"strength": doc["extra_metadata"]["requirement_strength"]})
|
||||
continue
|
||||
|
||||
if args.dry_run:
|
||||
results.append({"id": reg_id, "chars": len(text), "new": "?",
|
||||
"strength": doc["extra_metadata"]["requirement_strength"]})
|
||||
continue
|
||||
|
||||
if existing and existing > 0:
|
||||
print(f" SKIP — {existing} chunks already present. "
|
||||
f"Use Qdrant delete-by-filter before re-ingesting.")
|
||||
results.append({"id": reg_id, "chars": len(text), "new": 0,
|
||||
"strength": doc["extra_metadata"]["requirement_strength"]})
|
||||
continue
|
||||
|
||||
print(" Uploading with chunk_strategy='legal'...")
|
||||
result = upload_text_legal(
|
||||
text, doc["upload_filename"], doc["extra_metadata"]
|
||||
)
|
||||
new_chunks = result.get("chunks_count", 0)
|
||||
new_doc_id = result.get("document_id", "")
|
||||
print(f" -> {new_chunks} chunks (doc_id={new_doc_id})")
|
||||
|
||||
results.append({"id": reg_id, "chars": len(text), "new": new_chunks,
|
||||
"strength": doc["extra_metadata"]["requirement_strength"]})
|
||||
|
||||
if i < len(docs):
|
||||
time.sleep(2)
|
||||
|
||||
print("\n" + "=" * 70)
|
||||
print("SUMMARY")
|
||||
print("=" * 70)
|
||||
for r in results:
|
||||
print(f" {r['id']:<55} chars={r['chars']:<9} new={r['new']:<5} "
|
||||
f"strength={r['strength']}")
|
||||
total_new = sum(r["new"] for r in results if isinstance(r["new"], int))
|
||||
print(f"\nTotal new chunks: {total_new}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -22,6 +22,7 @@ import json
|
||||
import logging
|
||||
import time
|
||||
from collections import defaultdict
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
@@ -108,24 +109,37 @@ class BatchDedupRunner:
|
||||
self._progress_phase = ""
|
||||
self._progress_count = 0
|
||||
self._progress_total = 0
|
||||
self._since = None # set by run() when scoped run requested
|
||||
|
||||
async def run(
|
||||
self,
|
||||
dry_run: bool = False,
|
||||
hint_filter: str = None,
|
||||
since: datetime = None,
|
||||
) -> dict:
|
||||
"""Run the full batch dedup pipeline.
|
||||
|
||||
Args:
|
||||
dry_run: If True, compute stats but don't modify DB/Qdrant.
|
||||
hint_filter: If set, only process groups matching this hint prefix.
|
||||
since: If set, only process controls with created_at >= since.
|
||||
Useful for incremental dedup after single-document ingestion.
|
||||
|
||||
Returns:
|
||||
Stats dict with counts.
|
||||
"""
|
||||
start = time.monotonic()
|
||||
logger.info("BatchDedup starting (dry_run=%s, hint_filter=%s)",
|
||||
dry_run, hint_filter)
|
||||
logger.info("BatchDedup starting (dry_run=%s, hint_filter=%s, since=%s)",
|
||||
dry_run, hint_filter, since)
|
||||
|
||||
# Scoped runs reset checkpoint to avoid skipping new controls whose
|
||||
# control_id sorts before the stale last_id of a previous full run.
|
||||
self._since = since
|
||||
if since and not dry_run:
|
||||
self.db.execute(text(
|
||||
"DELETE FROM canonical_generation_jobs WHERE status = 'dedup_phase2_checkpoint'"
|
||||
))
|
||||
self.db.commit()
|
||||
|
||||
if not dry_run:
|
||||
await ensure_qdrant_collection(collection=self.collection)
|
||||
@@ -133,7 +147,7 @@ class BatchDedupRunner:
|
||||
# Phase 1: Intra-group dedup (same merge_group_hint)
|
||||
# Optimization: skip singleton groups (they're automatically masters)
|
||||
self._progress_phase = "phase1"
|
||||
groups = self._load_merge_groups(hint_filter)
|
||||
groups = self._load_merge_groups(hint_filter, since)
|
||||
self._progress_total = self.stats["total_controls"]
|
||||
|
||||
multi_groups = [(h, c) for h, c in groups if len(c) > 1]
|
||||
@@ -171,7 +185,7 @@ class BatchDedupRunner:
|
||||
logger.info("BatchDedup completed in %.1fs: %s", elapsed, self.stats)
|
||||
return self.stats
|
||||
|
||||
def _load_merge_groups(self, hint_filter: str = None) -> list:
|
||||
def _load_merge_groups(self, hint_filter: str = None, since: datetime = None) -> list:
|
||||
"""Load all Pass 0b controls grouped by merge_group_hint, largest first."""
|
||||
conditions = [
|
||||
"decomposition_method = 'pass0b'",
|
||||
@@ -184,6 +198,10 @@ class BatchDedupRunner:
|
||||
conditions.append("generation_metadata->>'merge_group_hint' LIKE :hf")
|
||||
params["hf"] = f"{hint_filter}%"
|
||||
|
||||
if since:
|
||||
conditions.append("created_at >= :since")
|
||||
params["since"] = since
|
||||
|
||||
where = " AND ".join(conditions)
|
||||
rows = self.db.execute(text(f"""
|
||||
SELECT id::text, control_id, title, objective,
|
||||
@@ -335,13 +353,15 @@ class BatchDedupRunner:
|
||||
"""
|
||||
logger.info("BatchDedup Phase 2: Cross-group pass starting...")
|
||||
|
||||
# Count total
|
||||
total_row = self.db.execute(text("""
|
||||
# Count total — respect scoped run if since is set
|
||||
since_clause = " AND created_at >= :since" if self._since else ""
|
||||
params = {"since": self._since} if self._since else {}
|
||||
total_row = self.db.execute(text(f"""
|
||||
SELECT COUNT(*) FROM canonical_controls
|
||||
WHERE decomposition_method = 'pass0b'
|
||||
AND release_state != 'duplicate'
|
||||
AND release_state != 'deprecated'
|
||||
""")).fetchone()
|
||||
AND release_state != 'deprecated'{since_clause}
|
||||
"""), params).fetchone()
|
||||
total = total_row[0] if total_row else 0
|
||||
|
||||
self._progress_total = total
|
||||
@@ -360,13 +380,16 @@ class BatchDedupRunner:
|
||||
last_control_id = checkpoint_row[0] if checkpoint_row else ""
|
||||
|
||||
if last_control_id:
|
||||
skip_row = self.db.execute(text("""
|
||||
skip_params = {"last_id": last_control_id}
|
||||
if self._since:
|
||||
skip_params["since"] = self._since
|
||||
skip_row = self.db.execute(text(f"""
|
||||
SELECT COUNT(*) FROM canonical_controls
|
||||
WHERE decomposition_method = 'pass0b'
|
||||
AND release_state != 'duplicate'
|
||||
AND release_state != 'deprecated'
|
||||
AND control_id <= :last_id
|
||||
"""), {"last_id": last_control_id}).fetchone()
|
||||
AND control_id <= :last_id{since_clause}
|
||||
"""), skip_params).fetchone()
|
||||
skipped = skip_row[0] if skip_row else 0
|
||||
self._progress_count = skipped
|
||||
logger.info("BatchDedup Cross-group: RESUMING from %s (skipping %d already processed)",
|
||||
@@ -382,17 +405,20 @@ class BatchDedupRunner:
|
||||
total, last_control_id or "beginning")
|
||||
|
||||
while True:
|
||||
rows = self.db.execute(text("""
|
||||
page_params = {"last_id": last_control_id, "page_size": DB_PAGE}
|
||||
if self._since:
|
||||
page_params["since"] = self._since
|
||||
rows = self.db.execute(text(f"""
|
||||
SELECT id::text, control_id, title,
|
||||
generation_metadata->>'merge_group_hint' as merge_group_hint
|
||||
FROM canonical_controls
|
||||
WHERE decomposition_method = 'pass0b'
|
||||
AND release_state != 'duplicate'
|
||||
AND release_state != 'deprecated'
|
||||
AND control_id > :last_id
|
||||
AND control_id > :last_id{since_clause}
|
||||
ORDER BY control_id
|
||||
LIMIT :page_size
|
||||
"""), {"last_id": last_control_id, "page_size": DB_PAGE}).fetchall()
|
||||
"""), page_params).fetchall()
|
||||
|
||||
if not rows:
|
||||
break
|
||||
|
||||
@@ -0,0 +1,860 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en" dir="ltr" class="h-100">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="MobileOptimized" content="width" />
|
||||
<meta name="HandheldFriendly" content="true" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<link rel="icon" href="/themes/custom/enisaweb/favicon.ico" type="image/png" />
|
||||
<link rel="alternate" type="application/rss+xml" title="Single Reporting Platform (SRP)" href="https://www.enisa.europa.eu/taxonomy/term/1317/feed" />
|
||||
<script>window.a2a_config=window.a2a_config||{};a2a_config.callbacks=[];a2a_config.overlays=[];a2a_config.templates={};</script>
|
||||
|
||||
<meta content="ENISA: Every day we experience the Information Society. Interconnected networks touch our everyday lives, at home and at work. It is therefore vital that computers, mobile phones, banking, and the Internet function, to support Europe’s digital economy. That is why ENISA is working with Cybersecurity for the EU and the Member States." name="DC.description">
|
||||
<meta name="description" content="ENISA is the EU agency dedicated to enhancing cybersecurity in Europe. They offer guidance, tools, and resources to safeguard citizens and businesses from cyber threats.">
|
||||
<meta name="keywords" content="Cybersecurity, EU, ENISA, computer security, Cyber Threats, EU Cyber Crisis, Incident Management, Market and Standards, Product Security, Security certification, Risk Management, Skills and competences, State of cybersecurity in the EU, Vulnerability Disclosure, Artificial Intelligence, Next Gen Technologies, Awareness, Cyber Hygiene, Digital Identity, Data Protection, Education and career path">
|
||||
<title>Single Reporting Platform (SRP) | ENISA</title>
|
||||
|
||||
<link rel="stylesheet" media="all" href="/sites/default/files/css/css_H8-PkuOemoNPxq-HW0ue4hGKWqBFO5KaLA29hyssQWk.css?delta=0&language=en&theme=enisaweb&include=eJxtj2sOwyAMgy9E4UhVCl6HRpKJ0FXd6Uf3lrY_kfPJkR1KqSnJFugp_KGqNBdPSLlpHSlGrSmrhLd6WCDJdYAgWplKvsJBstGKKUxkX9tcdKIyWNtKlvnDGWY0w5xpzFRG7pE0ds_JwqEndJMpw0flswp6qz_GX-TbEQxnmzXwo8olY7Vwn541LQVuRb-ZiSFLD6vsVww7GHYyvD68AUVHcCg" />
|
||||
<link rel="stylesheet" media="all" href="//cdnjs.cloudflare.com/ajax/libs/font-awesome/6.6.0/css/all.min.css" />
|
||||
<link rel="stylesheet" media="all" href="/sites/default/files/css/css_XfzkZLkUSSs_yRcqoRmh-VWG0krtdRIrQV-ENlV19ao.css?delta=2&language=en&theme=enisaweb&include=eJxtj2sOwyAMgy9E4UhVCl6HRpKJ0FXd6Uf3lrY_kfPJkR1KqSnJFugp_KGqNBdPSLlpHSlGrSmrhLd6WCDJdYAgWplKvsJBstGKKUxkX9tcdKIyWNtKlvnDGWY0w5xpzFRG7pE0ds_JwqEndJMpw0flswp6qz_GX-TbEQxnmzXwo8olY7Vwn541LQVuRb-ZiSFLD6vsVww7GHYyvD68AUVHcCg" />
|
||||
<link rel="stylesheet" media="all" href="/sites/default/files/css/css_gcXUcvuow4apg85qsW-WFQB8ls5BPBU3WeuPLmwnlqQ.css?delta=3&language=en&theme=enisaweb&include=eJxtj2sOwyAMgy9E4UhVCl6HRpKJ0FXd6Uf3lrY_kfPJkR1KqSnJFugp_KGqNBdPSLlpHSlGrSmrhLd6WCDJdYAgWplKvsJBstGKKUxkX9tcdKIyWNtKlvnDGWY0w5xpzFRG7pE0ds_JwqEndJMpw0flswp6qz_GX-TbEQxnmzXwo8olY7Vwn541LQVuRb-ZiSFLD6vsVww7GHYyvD68AUVHcCg" />
|
||||
|
||||
|
||||
|
||||
</head>
|
||||
<body class="path-taxonomy d-flex flex-column h-100">
|
||||
|
||||
|
||||
<div class="dialog-off-canvas-main-canvas d-flex flex-column h-100" data-off-canvas-main-canvas>
|
||||
|
||||
|
||||
|
||||
<header>
|
||||
<p class="sr-only"><a href="#main-content" accesskey="M">Go to the main content</a></p>
|
||||
|
||||
<div class="navbar navbar-expand-lg navbar-dark text-light bg-primary header">
|
||||
<div class="container logo-menu-wrapper d-flex">
|
||||
|
||||
<div class="region region-nav-branding">
|
||||
<div id="block-enisaweb-branding" class="block block-system block-system-branding-block">
|
||||
|
||||
|
||||
<div class="navbar-brand d-flex align-items-center">
|
||||
|
||||
<a href="/" title="Home" rel="home" class="site-logo d-block">
|
||||
<img src="/sites/default/files/enisa-logo.svg" alt="Home" fetchpriority="high" />
|
||||
</a>
|
||||
|
||||
<div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
<nav class="collapse navbar-collapse justify-content-end main-menu" id="navbarSupportedContent" role="navigation" aria-label="Main menu and Search box">
|
||||
<div class="region region-nav-main">
|
||||
<div id="block-enisaweb-mainnavigation-2" class="block block-we-megamenu block-we-megamenu-blockmain">
|
||||
|
||||
|
||||
<div class="region-we-mega-menu">
|
||||
<a href="javascript:" class="navbar-toggle collapsed" aria-label="Open/close menu" aria-controls="mainMenuResponsive" name="menu-button" role="button">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</a>
|
||||
<nav class="main navbar navbar-default navbar-we-mega-menu mobile-collapse hover-action" data-menu-name="main" data-block-theme="enisaweb" data-style="Default" data-animation="None" data-delay="" data-duration="" data-autoarrow="" data-alwayshowsubmenu="" data-action="hover" data-mobile-collapse="0" aria-label="ENISA main menu" id="mainMenuResponsive">
|
||||
<div class="container-fluid">
|
||||
<ul class="we-mega-menu-ul nav nav-tabs">
|
||||
<li class="we-mega-menu-li justify dropdown-menu" title="" data-level="0" data-element-type="" data-id="320e2c86-310b-4f7b-a4a9-188df34c3e43" data-submenu="1" data-hide-sub-when-collapse="" data-group="0" data-caption="" data-alignsub="justify" data-target="" data-icon="" >
|
||||
<span class="we-megamenu-nolink">
|
||||
Topics</span>
|
||||
<div class="we-mega-menu-submenu" data-element-type="we-mega-menu-submenu" data-submenu-width="" data-class="" style="width: px">
|
||||
<div class="we-mega-menu-submenu-inner">
|
||||
<div class="we-mega-menu-row" data-element-type="we-mega-menu-row" data-custom-row="1">
|
||||
<div class="we-mega-menu-col span3" data-element-type="we-mega-menu-col" data-width="3" data-block="enisaweb_topics" data-blocktitle="0" data-hidewhencollapse="" data-class="">
|
||||
<div class="type-of-block"><div class="block-inner"><div id="block-enisaweb-topics" class="block block-block-content block-block-contentec3f1f7d-35d4-4776-a03c-7f97a2fcfc8f">
|
||||
|
||||
<p class="title"><a href="/topics" target="_self" title="Access to All Topics page">Topics</a></p>
|
||||
|
||||
|
||||
<div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>Learn more about the topics</p>
|
||||
<p><a class="button" href="/topics" data-entity-type="node" data-entity-uuid="5f4db810-e19d-49a4-baaa-86ec782eb91e" data-entity-substitution="canonical" title="Topics">Access</a></p>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div></div>
|
||||
|
||||
</div>
|
||||
<div class="we-mega-menu-col span3" data-element-type="we-mega-menu-col" data-width="3" data-block="enisaweb_views_block__topics_tax_block_1" data-blocktitle="0" data-hidewhencollapse="" data-class="">
|
||||
<div class="type-of-block"><div class="block-inner"><div class="views-element-container block block-views block-views-blocktopics-tax-block-1" id="block-enisaweb-views-block-topics-tax-block-1">
|
||||
|
||||
<p class="title"><a href="/audience/national-eu-authorities" target="_self" title="Access to For National / EU authorities page">For National / EU authorities</a></p>
|
||||
|
||||
<div data-block="nav_main"><div class="view view-topics-tax view-id-topics_tax view-display-id-block_1 js-view-dom-id-a9128a4de087e3f9960a31f862089b7e6946eeb6d529b846901e74f4ba95f84d">
|
||||
|
||||
|
||||
|
||||
<div class="view-content">
|
||||
<div class="item-list">
|
||||
|
||||
<ul>
|
||||
|
||||
<li><a href="/topics/cyber-threats" hreflang="en">Cyber Threats</a></li>
|
||||
<li><a href="/topics/eu-incident-response-and-cyber-crisis-management" hreflang="en">EU incident response and cyber crisis management</a></li>
|
||||
<li><a href="/topics/market" hreflang="en">Market</a></li>
|
||||
<li><a href="/topics/product-security-and-certification" hreflang="en">Product Security and Certification</a></li>
|
||||
<li><a href="/topics/risk-management" hreflang="en">Risk Management</a></li>
|
||||
<li><a href="/topics/skills-and-competences" hreflang="en">Skills and competences</a></li>
|
||||
<li><a href="/topics/state-of-cybersecurity-in-the-eu" hreflang="en">State of cybersecurity in the EU</a></li>
|
||||
<li><a href="/topics/vulnerability-disclosure" hreflang="en">Vulnerability Disclosure</a></li>
|
||||
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div></div></div>
|
||||
|
||||
</div>
|
||||
<div class="we-mega-menu-col span3" data-element-type="we-mega-menu-col" data-width="3" data-block="enisaweb_views_block__topics_tax_block_2" data-blocktitle="0" data-hidewhencollapse="" data-class="">
|
||||
<div class="type-of-block"><div class="block-inner"><div class="views-element-container title block block-views block-views-blocktopics-tax-block-2" id="block-enisaweb-views-block-topics-tax-block-2">
|
||||
|
||||
<p class="title"><a href="/audience/private-sector" target="_self" title="Access to Private Sector page">Private Sector</a></p>
|
||||
|
||||
<div data-block="nav_main"><div class="view view-topics-tax view-id-topics_tax view-display-id-block_2 js-view-dom-id-ca2d4c425826cfbc7b36885c48a7a8c805021f1f25dec545fefac51557010f38">
|
||||
|
||||
|
||||
|
||||
<div class="view-content">
|
||||
<div class="item-list">
|
||||
|
||||
<ul>
|
||||
|
||||
<li><a href="/topics/artificial-intelligence-and-next-gen-technologies" hreflang="en">Artificial Intelligence and Next Gen Technologies</a></li>
|
||||
<li><a href="/topics/awareness-and-cyber-hygiene" hreflang="en">Awareness and Cyber Hygiene</a></li>
|
||||
<li><a href="/topics/certification-and-standards" hreflang="en">Certification and Standards</a></li>
|
||||
<li><a href="/topics/cyber-threats" hreflang="en">Cyber Threats</a></li>
|
||||
<li><a href="/topics/cybersecurity-of-critical-sectors" hreflang="en">Cybersecurity of Critical Sectors</a></li>
|
||||
<li><a href="/topics/digital-identity-and-data-protection" hreflang="en">Digital Identity and Data Protection</a></li>
|
||||
<li><a href="/topics/incident-management" hreflang="en">Incident management</a></li>
|
||||
<li><a href="/topics/risk-management" hreflang="en">Risk Management</a></li>
|
||||
<li><a href="/topics/skills-and-competences-for-companies" hreflang="en">Skills and competences (for companies)</a></li>
|
||||
<li><a href="/topics/vulnerability-disclosure" hreflang="en">Vulnerability Disclosure</a></li>
|
||||
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div></div></div>
|
||||
|
||||
</div>
|
||||
<div class="we-mega-menu-col span3" data-element-type="we-mega-menu-col" data-width="3" data-block="enisaweb_views_block__topics_tax_block_3" data-blocktitle="0" data-hidewhencollapse="" data-class="">
|
||||
<div class="type-of-block"><div class="block-inner"><div class="views-element-container title block block-views block-views-blocktopics-tax-block-3" id="block-enisaweb-views-block-topics-tax-block-3">
|
||||
|
||||
<p class="title"><a href="/audience/citizens" target="_self" title="Access to Citizens page">Citizens</a></p>
|
||||
|
||||
<div data-block="nav_main"><div class="view view-topics-tax view-id-topics_tax view-display-id-block_3 js-view-dom-id-b69cbf795aad827e795977b39f2678f43cdc75c6765696171efc1a9faca05ccd">
|
||||
|
||||
|
||||
|
||||
<div class="view-content">
|
||||
<div class="item-list">
|
||||
|
||||
<ul>
|
||||
|
||||
<li><a href="/topics/cyber-hygiene" hreflang="en">Cyber Hygiene</a></li>
|
||||
<li><a href="/topics/cyber-incident-awareness" hreflang="en">Cyber Incident Awareness</a></li>
|
||||
<li><a href="/topics/education-and-career-path" hreflang="en">Education and career path</a></li>
|
||||
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div></div></div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</li><li class="we-mega-menu-li" title="Access to all publications" data-level="0" data-element-type="" data-id="2d24a690-803d-44ad-8d7e-8cf0eb7d0e40" data-submenu="0" data-hide-sub-when-collapse="" data-group="0" data-caption="" data-alignsub="" data-target="_self" data-icon="" >
|
||||
<a class="we-mega-menu-li" title="" href="/publications" target="_self">
|
||||
Publications </a>
|
||||
|
||||
</li><li class="we-mega-menu-li dropdown-menu" title="" data-level="0" data-element-type="" data-id="54b069d8-f99f-45e7-9d50-9345e82ad14f" data-submenu="1" data-hide-sub-when-collapse="" data-group="0" data-caption="" data-alignsub="" data-target="" data-icon="" >
|
||||
<span class="we-megamenu-nolink">
|
||||
Newsroom & Events</span>
|
||||
<div class="we-mega-menu-submenu" data-element-type="we-mega-menu-submenu" data-submenu-width="" data-class="" style="width: px">
|
||||
<div class="we-mega-menu-submenu-inner">
|
||||
<div class="we-mega-menu-row" data-element-type="we-mega-menu-row" data-custom-row="1">
|
||||
<div class="we-mega-menu-col span4" data-element-type="we-mega-menu-col" data-width="4" data-block="enisaweb_news" data-blocktitle="0" data-hidewhencollapse="" data-class="">
|
||||
<div class="type-of-block"><div class="block-inner"><div id="block-enisaweb-news" class="block block-block-content block-block-content00f65cb0-e1b0-41a1-8a0d-00ce6ced0f0e">
|
||||
|
||||
<p class="title"><a href="/news" target="_self" title="Access to All News">News</a></p>
|
||||
|
||||
|
||||
<div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>Cybersecurity in focus: News & updates from ENISA</p>
|
||||
<p><a class="button" href="/news" data-entity-type="node" data-entity-uuid="088dd847-33ff-4c8e-b094-f0d38a66bbdd" data-entity-substitution="canonical" title="News">Access</a></p>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div></div>
|
||||
|
||||
</div>
|
||||
<div class="we-mega-menu-col span4" data-element-type="we-mega-menu-col" data-width="4" data-block="enisaweb_events" data-blocktitle="0" data-hidewhencollapse="" data-class="">
|
||||
<div class="type-of-block"><div class="block-inner"><div id="block-enisaweb-events" class="block block-block-content block-block-contentb3c06780-f342-437d-9828-54fe86eb9786">
|
||||
|
||||
<p class="title"><a href="/events" target="_self" title="Access to All Events">Events</a></p>
|
||||
|
||||
|
||||
<div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>Cybersecurity in practice: Events & Workshops by ENISA</p>
|
||||
<p><a class="button" href="/events" data-entity-type="node" data-entity-uuid="34f66c6d-0f07-4977-85e3-e8578a59d59c" data-entity-substitution="canonical" title="Events">Access</a></p>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div></div>
|
||||
|
||||
</div>
|
||||
<div class="we-mega-menu-col span4" data-element-type="we-mega-menu-col" data-width="4" data-block="enisaweb_pressoffice" data-blocktitle="0" data-hidewhencollapse="" data-class="">
|
||||
<div class="type-of-block"><div class="block-inner"><div id="block-enisaweb-pressoffice" class="block block-system block-system-menu-blockpress-office">
|
||||
|
||||
<p class="title"><a href="/press-office" target="_self" title="Access to Press office page">Press office</a></p>
|
||||
|
||||
|
||||
<ul data-block="nav_main" class="nav navbar-nav">
|
||||
<li class="nav-item">
|
||||
<a href="/press-office/corporate-identity" class="nav-link" data-drupal-link-system-path="node/11030">Corporate identity</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/press-office/cybersecurity-material" class="nav-link" data-drupal-link-system-path="node/11031">Cybersecurity material</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/form/seek-an-expert" class="nav-link" data-drupal-link-system-path="webform/seek_an_expert">Seek an expert or Request a speaker</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
|
||||
|
||||
</div></div></div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</li><li class="we-mega-menu-li dropdown-menu" title="" data-level="0" data-element-type="" data-id="eaba29ce-6446-4da3-930a-c3d97495f00b" data-submenu="1" data-hide-sub-when-collapse="" data-group="0" data-caption="" data-alignsub="" data-target="" data-icon="" >
|
||||
<span class="we-megamenu-nolink">
|
||||
About</span>
|
||||
<div class="we-mega-menu-submenu" data-element-type="we-mega-menu-submenu" data-submenu-width="" data-class="" style="width: px">
|
||||
<div class="we-mega-menu-submenu-inner">
|
||||
<div class="we-mega-menu-row" data-element-type="we-mega-menu-row" data-custom-row="1">
|
||||
<div class="we-mega-menu-col span4" data-element-type="we-mega-menu-col" data-width="4" data-block="enisaweb_whatwedosubmenubutton_2" data-blocktitle="0" data-hidewhencollapse="" data-class="">
|
||||
<div class="type-of-block"><div class="block-inner"><div id="block-enisaweb-whatwedosubmenubutton-2" class="block block-block-content block-block-contente0a96203-1eca-4991-9533-33064cace452">
|
||||
|
||||
<p class="title"><a href="/about-enisa/what-we-do" target="_self" title="Access to What we do page">What we do</a></p>
|
||||
|
||||
|
||||
<div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>Achieving a high common level of cybersecurity across Europe</p>
|
||||
<p><a class="button" href="/about-enisa/what-we-do" data-entity-type="node" data-entity-uuid="0619d110-d4dc-4f6b-aa0d-9a56252da07d" data-entity-substitution="canonical" title="What we do">Access</a></p>
|
||||
</div>
|
||||
|
||||
</div></div></div>
|
||||
|
||||
</div>
|
||||
<div class="we-mega-menu-col span4" data-element-type="we-mega-menu-col" data-width="4" data-block="enisaweb_whatwedosubmenu_2" data-blocktitle="0" data-hidewhencollapse="" data-class="">
|
||||
<div class="type-of-block"><div class="block-inner"><div id="block-enisaweb-whatwedosubmenu-2" class="block block-block-content block-block-content99599d89-56f7-43f1-907b-72614045018d">
|
||||
|
||||
<p class="title"><a href="/about-enisa/who-we-are" target="_self" title="Access to Who we are page">Who we are</a></p>
|
||||
|
||||
|
||||
<div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>Towards a Trusted and Cyber Secure Europe </p>
|
||||
<p><a class="button" href="/about-enisa/who-we-are" data-entity-type="node" data-entity-uuid="3c46c210-298c-472c-b871-b167dfe2642c" data-entity-substitution="canonical" title="Who we are">Access</a></p>
|
||||
</div>
|
||||
|
||||
</div></div></div>
|
||||
|
||||
</div>
|
||||
<div class="we-mega-menu-col span4" data-element-type="we-mega-menu-col" data-width="4" data-block="enisaweb_transparency" data-blocktitle="0" data-hidewhencollapse="" data-class="">
|
||||
<div class="type-of-block"><div class="block-inner"><div id="block-enisaweb-transparency" class="block block-system block-system-menu-blocktransparency">
|
||||
|
||||
<p class="title"><a href="/about-enisa/How-we-work" target="_self" title="Access to How we work page">How we work</a></p>
|
||||
|
||||
|
||||
<ul data-block="nav_main" class="nav navbar-nav">
|
||||
<li class="nav-item">
|
||||
<a href="/about-enisa/accounting-finance/accounting-finance" class="nav-link" data-drupal-link-system-path="node/17339">Accounting and Finance</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/about-enisa/procedures-and-policies" class="nav-link dropdown-toggle" data-drupal-link-system-path="node/17360">Policies and Procedures</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/about-enisa/data-protection/data-protection" class="nav-link dropdown-toggle" data-drupal-link-system-path="node/17346">Data Protection</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/about-enisa/transparency" class="nav-link dropdown-toggle" data-drupal-link-system-path="node/17404">Transparency</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/enisa-a-climate-neutral-agency" class="nav-link" data-drupal-link-system-path="node/18893">ENISA, a climate neutral agency </a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</div></div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
<div class="we-mega-menu-row" data-element-type="we-mega-menu-row" data-custom-row="1">
|
||||
<div class="we-mega-menu-col span12 transversal-menu" data-element-type="we-mega-menu-col" data-width="12" data-block="enisaweb_transversaloptionsofaboutmenu" data-blocktitle="0" data-hidewhencollapse="" data-class="transversal-menu">
|
||||
<div class="type-of-block"><div class="block-inner"><div id="block-enisaweb-transversaloptionsofaboutmenu" class="block block-system block-system-menu-blocktransversal-options-about-menu">
|
||||
|
||||
|
||||
|
||||
<ul data-block="nav_main" class="nav navbar-nav">
|
||||
<li class="nav-item">
|
||||
<a href="/about-enisa/international-cooperation" class="nav-link" data-drupal-link-system-path="node/19377">International Cooperation</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</div></div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</li><li class="we-mega-menu-li dropdown-menu" title="" data-level="0" data-element-type="" data-id="44fbec5d-1aaf-4cb3-b06b-a79b571ad50c" data-submenu="1" data-hide-sub-when-collapse="" data-group="0" data-caption="" data-alignsub="" data-target="" data-icon="" >
|
||||
<span class="we-megamenu-nolink">
|
||||
Working with us</span>
|
||||
<div class="we-mega-menu-submenu" data-element-type="we-mega-menu-submenu" data-submenu-width="" data-class="" style="width: px">
|
||||
<div class="we-mega-menu-submenu-inner">
|
||||
<div class="we-mega-menu-row" data-element-type="we-mega-menu-row" data-custom-row="1">
|
||||
<div class="we-mega-menu-col span4" data-element-type="we-mega-menu-col" data-width="4" data-block="enisaweb_workwithenisa" data-blocktitle="0" data-hidewhencollapse="" data-class="">
|
||||
<div class="type-of-block"><div class="block-inner"><div id="block-enisaweb-workwithenisa" class="block block-block-content block-block-content86d5ce2f-abc2-4160-afda-442fe2c68258">
|
||||
|
||||
<p class="title"><a href="/working-with-us/working-for-enisa" target="_self" title="Access to Working for ENISA page">Working for ENISA</a></p>
|
||||
|
||||
|
||||
<div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>Explore the benefits of working for ENISA</p>
|
||||
<p><a class="button" href="/working-with-us/working-for-enisa" data-entity-type="node" data-entity-uuid="f10da531-46ad-49f6-84cb-0770c69d839e" data-entity-substitution="canonical" title="Working for ENISA">Access</a></p>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div></div>
|
||||
|
||||
</div>
|
||||
<div class="we-mega-menu-col span8" data-element-type="we-mega-menu-col" data-width="8" data-block="enisaweb_workingwithus" data-blocktitle="0" data-hidewhencollapse="" data-class="">
|
||||
<div class="type-of-block"><div class="block-inner"><div id="block-enisaweb-workingwithus" class="title block block-system block-system-menu-blockworking-with-us">
|
||||
|
||||
<p class="title"><a href="/work-with-us" target="_self" title="Access to Working with us page">Working with us</a></p>
|
||||
|
||||
|
||||
<ul data-block="nav_main" class="nav navbar-nav">
|
||||
<li class="nav-item">
|
||||
<a href="/careers" class="nav-link" data-drupal-link-system-path="node/12487">Careers</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/working-with-us/procurement" class="nav-link" data-drupal-link-system-path="node/17421">Procurement</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/working-with-us/ad-hoc-working-groups-calls" class="nav-link" data-drupal-link-system-path="node/17422">Ad hoc working groups</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
|
||||
|
||||
</div></div></div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</nav>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<p class="btn-search">
|
||||
<a href="javascript:" title="Show the search field">Search</a>
|
||||
</p>
|
||||
<div class="search-form-wrapper" aria-label="Search field">
|
||||
<div class="region region-nav-additional">
|
||||
<div id="block-enisaweb-search" class="search-block-form block block-search container-inline" data-drupal-selector="search-block-form" id="block-enisaweb-search-form" role="search">
|
||||
|
||||
|
||||
<form data-block="nav_additional" action="/search" method="get" id="search-block-form" accept-charset="UTF-8">
|
||||
<div class="js-form-item form-item js-form-type-search form-type-search js-form-item-keys form-item-keys form-no-label">
|
||||
<label class="visually-hidden" for="edit-keys">Search</label>
|
||||
<input class="form-search form-control" title="Enter the terms you wish to search for." data-drupal-selector="edit-keys" type="search" id="edit-keys" name="keys" value="" size="15" maxlength="128">
|
||||
</div>
|
||||
<div class="form-actions js-form-wrapper form-wrapper" data-drupal-selector="edit-actions" id="edit-actions-search">
|
||||
<button class="button js-form-submit form-submit btn-enisa btn-primary" data-drupal-selector="edit-submit" type="submit" id="edit-submit" value="Search">Search</button>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<p class="btn-close-search"><a href="javascript:" title="Hide the search field">Close</a></p>
|
||||
</div>
|
||||
</nav>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<main role="main" class="container-full image-banner">
|
||||
<div class="title-breadcrumbs container">
|
||||
<div id="block-enisaweb-page-title" class="block block-core block-page-title-block">
|
||||
|
||||
|
||||
|
||||
<h1>Single Reporting Platform (SRP)</h1>
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
<div class="region region-breadcrumb">
|
||||
<div id="block-enisaweb-breadcrumbs" class="block block-system block-system-breadcrumb-block">
|
||||
|
||||
|
||||
|
||||
<nav aria-label="breadcrumb">
|
||||
<p id="system-breadcrumb" class="visually-hidden">Breadcrumb</p>
|
||||
<ol class="breadcrumb">
|
||||
<li class="breadcrumb-item">
|
||||
<a href="/" aria-label="Access to ">Home</a>
|
||||
</li>
|
||||
<li class="breadcrumb-item">
|
||||
<a href="/topics" aria-label="Access to ">Topics</a>
|
||||
</li>
|
||||
<li class="breadcrumb-item">
|
||||
<a href="/topics/product-security-and-certification" aria-label="Access to ">Product Security and Certification</a>
|
||||
</li>
|
||||
<li class="breadcrumb-item">
|
||||
Single Reporting Platform (SRP)
|
||||
</li>
|
||||
</ol>
|
||||
</nav>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
<div class="main container container-taxonomy">
|
||||
<a id="main-content" tabindex="-1"></a>
|
||||
<div class="container">
|
||||
<div data-drupal-messages-fallback class="hidden"></div>
|
||||
|
||||
|
||||
|
||||
<div class="row g-0 publications-list-section">
|
||||
|
||||
<div class="sidebar-first order-lg-1 col-12 col-lg-3">
|
||||
<div id="block-enisaweb-topictaxonomymenublock" class="block block-enisa-path-block block-topic-taxonomy-menu-block">
|
||||
|
||||
|
||||
<h2>Subtopics</h2>
|
||||
|
||||
<div class="item-list">
|
||||
<ul class="submenu">
|
||||
<li>
|
||||
<a href="https://www.enisa.europa.eu//topics/product-security-and-certification/single-reporting-platform-srp" hreflang="en">Single Reporting Platform (SRP)</a>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
<div class="content-wrapper order-lg-2 col-12 col-lg-9">
|
||||
<div id="block-enisaweb-content" class="block block-system block-system-main-block">
|
||||
|
||||
|
||||
<div class="views-element-container"><div class="view view-taxonomy-term view-id-taxonomy_term view-display-id-page_1 js-view-dom-id-fe094e9a97c4c6ce1116eacf5ed98f737017da96312d42131b1bb835e76fa9cb">
|
||||
|
||||
|
||||
<div class="view-header">
|
||||
|
||||
<div id="taxonomy-term-1317" class="taxonomy-term vocabulary-topics">
|
||||
<div class="content-description row">
|
||||
<div class="col-md-12 col-lg-12 tax-description">
|
||||
<div class="quote-wrapper">
|
||||
<p>The Cyber Resilience Act (CRA) introduces the Single Reporting Platform (SRP) for cybersecurity incident reporting in the EU Digital Single Market.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="row">
|
||||
<div class="col-lg-12 tax-content">
|
||||
|
||||
<div class="clearfix text-formatted field field--name-field-body field--type-text-with-summary field--label-hidden field__item"><p>The Single Reporting Platform (SRP) provided for in the Cyber Resilience Act (CRA) shall become a technical tool to use for the reporting of actively exploited vulnerabilities and incidents impacting products with digital elements operating in the EU Digital Single Market. </p>
|
||||
<p>The SRP will be used by CSIRTs and manufacturers for mandatory reporting and could be used by any natural/legal persons for voluntary reporting.</p>
|
||||
<p>The CRA mandates manufacturers of products with digital elements to report actively exploited vulnerabilities and severe incidents having an impact on the security of the product as of 11 September 2026 onwards using the Single Reporting Platform. Throughout 2025 and 2026, ENISA is undertaking a number of necessary steps to support the successful implementation of the platform.</p>
|
||||
<p>The CRA brings transparency to the vulnerability disclosure processes and strengthens how EU CSIRTs can mitigate risks stemming from vulnerabilities. </p>
|
||||
<p>Further information: <a href="https://eur-lex.europa.eu/eli/reg/2024/2847/oj/eng">Regulation - 2024/2847 - EN - EUR-Lex</a></p>
|
||||
<h3>Frequently Asked Questions</h3>
|
||||
<p>This is a collection of frequently asked questions on Cyber Resilience Act Single Reporting Platform (CRA SRP). Document is intended for publication on ENISA website and to be updated during implementation of CRA SRP</p>
|
||||
<p>Please see also information about CRA reporting <a href="https://digital-strategy.ec.europa.eu/en/policies/cra-reporting">https://digital-strategy.ec.europa.eu/en/policies/cra-reporting</a> in particular FAQ file there <a href="https://ec.europa.eu/newsroom/dae/redirection/document/122331">https://ec.europa.eu/newsroom/dae/redirection/document/122331</a></p>
|
||||
<dl class="ckeditor-accordion">
|
||||
<dt>What is the Cyber Resilience Act’s Single Reporting Platform (CRA SRP)?</dt>
|
||||
<dd>
|
||||
<p class="text-align-justify">The CRA SRP is an electronic system designed to simplify the reporting obligations for manufacturers under the Cyber Resilience Act. It allows for manufacturers to report actively exploited vulnerabilities and severe incidents having an impact on the security of products with digital elements only once, rather than having to notify multiple national authorities individually.</p>
|
||||
</dd>
|
||||
<dt>Who is responsible for establishing and managing the platform?</dt>
|
||||
<dd>
|
||||
<p>ENISA is tasked with establishing, managing, and maintaining the day-to-day operations of the CRA SRP. ENISA must also ensure the platform's security and implement appropriate technical and organisational measures to protect the information submitted.</p>
|
||||
</dd>
|
||||
<dt>When will the Single Reporting Platform be operational?</dt>
|
||||
<dd>
|
||||
<p>The platform is scheduled to be operational by 11 September 2026. This coincides with the date when the <strong>mandatory reporting</strong> obligations for manufacturers officially enter into application (art.14 of Cyber Resilience Act). A testing period is expected to take place before this date.</p>
|
||||
</dd>
|
||||
<dt>What must be reported via the platform?</dt>
|
||||
<dd>
|
||||
<p>Manufacturers must use the platform to notify two specific types of events:</p>
|
||||
<ul>
|
||||
<li><strong>Actively Exploited Vulnerabilities:</strong> Vulnerabilities in products with digital elements that are known to be currently exploited by a malicious actor.</li>
|
||||
<li><strong>Severe Incidents:</strong> Incidents that have a severe impact on the security of the product with digital elements (e.g., compromising availability, authenticity, integrity, or confidentiality); the criteria for severity are defined in Article 14(5).</li>
|
||||
</ul>
|
||||
</dd>
|
||||
<dt>What else can be reported in the platform? </dt>
|
||||
<dd>
|
||||
<p>The platform will also offer functionality to allow voluntary reporting. Any natural or legal person may notify on a voluntary basis: </p>
|
||||
<ul>
|
||||
<li><strong>Vulnerabilities </strong>contained in a product with digital elements;</li>
|
||||
<li><strong>Cyber threats</strong> that could affect the risk profile of a product with digital elements;</li>
|
||||
<li><strong>Incidents</strong> having an impact on the security of a product;</li>
|
||||
<li><strong>Near misses</strong> that could have resulted in an incident.</li>
|
||||
</ul>
|
||||
</dd>
|
||||
<dt>What are the deadlines for reporting?</dt>
|
||||
<dd>
|
||||
<p>Manufacturers must adhere to a multi-stage reporting timeline via the platform:</p>
|
||||
<ul>
|
||||
<li><strong>Early Warning: </strong>Without undue delay and in any case within <strong>24 hours </strong>of becoming aware of the vulnerability or incident.\</li>
|
||||
<li><strong>Vulnerability/Incident Notification: </strong>Without undue delay and in any case within <strong>72 hours</strong> of becoming aware, providing general information and an initial assessment.</li>
|
||||
<li><strong>Final Report:</strong>
|
||||
<ul>
|
||||
<li>For <strong>vulnerabilities</strong>: No later than <strong>14 days</strong> after a corrective measure (e.g., patch) is available.</li>
|
||||
<li>For <strong>severe incidents</strong>: Within <strong>1 month</strong> after the initial notification.<br> </li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</dd>
|
||||
<dt>How does the Single Reporting Platform operate?</dt>
|
||||
<dd>
|
||||
<p>Manufacturers submit notifications electronically through the platform, which automatically routes them to the designated CSIRT coordinator (based on the manufacturer's main establishment) and ENISA simultaneously. The CSIRT then disseminates the information without delay to other relevant CSIRTs in Member States where the product is available, and to market surveillance authorities as needed. For sensitive reports, dissemination may be delayed on security grounds, with ENISA informed and able to recommend broader sharing if risks are systemic. The platform incorporates security measures to protect confidentiality. </p>
|
||||
</dd>
|
||||
<dt>How do I know what is my designated CSIRT?</dt>
|
||||
<dd>
|
||||
<p>Your designated CSIRT is determined by your location of establishment:</p>
|
||||
<p>If you are established in the EU: Your designated CSIRT is the national CSIRT designated as the coordinator in the Member State where you have your main establishment. (please see CRA Art 14(7) for more details)</p>
|
||||
<p>If you are NOT established in the EU: Your designated CSIRT is the one designated as coordinator in the Member State where your authorised representative is established. (please see CRA Art 14(7) for more details)</p>
|
||||
</dd>
|
||||
<dt>What are the responsibilities of key entities involved with the CRA SRP?</dt>
|
||||
<dd>
|
||||
<ul>
|
||||
<li>Manufacturers: Submit timely notifications and comply with the other obligations established by the CRA. </li>
|
||||
<li>ENISA: Manages the platform, processes reports, prepares biennial trend reports (first due within 24 months of the reporting obligations starting), operates a helpdesk (especially for SMEs), and discloses fixed vulnerabilities to the European Vulnerability Database.</li>
|
||||
<li>CSIRTs Designated as Coordinators: Receive and assess reports, decide on dissemination delays, inform market surveillance authorities and the public if necessary, and provide helpdesk support alongside ENISA.</li>
|
||||
<li>European Commission: Adopts delegated and implementing acts (e.g., for delay criteria and report formats), evaluates the platform's effectiveness, and supports coordination of enforcement activities.</li>
|
||||
<li>Market Surveillance Authorities: Receive disseminated information and enforce compliance, such as through investigations or corrective actions.</li>
|
||||
</ul>
|
||||
</dd>
|
||||
<dt>Who receives the reports submitted to the platform?</dt>
|
||||
<dd>
|
||||
<p>As a general rule, when a manufacturer submits a report to the CRA SRP, it is simultaneously notified to:</p>
|
||||
<ul>
|
||||
<li>The <strong>CSIRT</strong> (Computer Security Incident Response Team) designated as the coordinator in the Member State where the manufacturer is established.</li>
|
||||
<li><strong>ENISA </strong>(unless particularly exceptional circumstances apply).</li>
|
||||
</ul>
|
||||
<p>The CSIRT designated as coordinator that initially receives the notification is then responsible for disseminating it without delay to other relevant CSIRTs across the EU via the platform.</p>
|
||||
</dd>
|
||||
<dt>Can the dissemination of a report be delayed or withheld?</dt>
|
||||
<dd>
|
||||
<p>Yes. In exceptional circumstances, the receiving CSIRT may decide to delay or withhold the dissemination of a notification to other Member States. This is strictly limited to cases where immediate dissemination is justified on security related grounds (e.g., if spreading the information would pose an even greater security risk).</p>
|
||||
<p>The European Commission adopted a delegated act on <strong>11 December 2025</strong> to further specify the terms and conditions for applying these grounds. [<a href="https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=PI_COM:C(2025)8407">https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=PI_COM:C(2025)8407</a>] </p>
|
||||
<p>In particularly exceptional circumstances, ENISA will not receive the full content of the 72-hour notification. This is only the case where, in the 72-hour notification, the manufacturer actively marks that at least one of the conditions listed in points (a) to (c) of Article 16(2) applies. In such case, ENISA only receives partial information, until the receiving CSIRT discloses the full notification.</p>
|
||||
</dd>
|
||||
<dt>How does the platform ensure security?</dt>
|
||||
<dd>
|
||||
<p>ENISA is legally required to take appropriate measures to manage risks to the platform's security and must notify the CSIRTs Network and the Commission of any security incidents affecting the platform itself.</p>
|
||||
</dd>
|
||||
<dt>How is the CSIRTs network involved?</dt>
|
||||
<dd>
|
||||
<p>As provided in CRA Article 16 ENISA is engaging the CSIRTs Network in development and future testing of the CRA SRP.</p>
|
||||
</dd>
|
||||
</dl>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<p class="btn-back back-to-tax"><a href="https://www.enisa.europa.eu/taxonomy/term/519">Back to main topic</a></p>
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
</main>
|
||||
|
||||
<section class="subscribe-section" role="complementary">
|
||||
<div class="container">
|
||||
<div class="row">
|
||||
<div class="col-md-3 col-lg-3 subscribe-image"></div>
|
||||
<div class="col-md-9 col-lg-9 subscribe-wrapper">
|
||||
<h2>Subscribe</h2>
|
||||
<p><strong>Stay updated with ENISA!</strong> Sign up for email alerts on publications, events, vacancies,
|
||||
and more.</p>
|
||||
<p><a href="/alertservice" class="btn-all left">Sign up now</a></p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<footer class="mt-auto enisa-footer">
|
||||
<div class="container">
|
||||
<div class="footer container">
|
||||
<div class="row">
|
||||
<div class="col-md-12 col-lg-3">
|
||||
<div id="block-enisaweb-enisalogos">
|
||||
|
||||
|
||||
|
||||
<div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item">
|
||||
<div class="enisa-logo">
|
||||
<img src="/themes/custom/enisaweb/images/enisa-logo-white.svg"
|
||||
alt="ENISA, European Union Agency for Cybersecurity" width="220" height="150"
|
||||
class="align-left" loading="lazy">
|
||||
<p><em>A Trusted and Cyber Secure Europe</em></p>
|
||||
</div>
|
||||
<div class="agencies-network-logo">
|
||||
<img src="/themes/custom/enisaweb/images/agencies-network.png"
|
||||
alt="Agencies network logo" width="39" height="36" class="align-left"
|
||||
loading="lazy">
|
||||
<p><a href="https://agencies-network.europa.eu/index_en">EU Agencies Network</a></p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<div id="block-enisaweb-socialmedialinks" class="block-social-media-links block block-social-media-links-block">
|
||||
|
||||
<p>Follow us on</p>
|
||||
|
||||
|
||||
|
||||
<ul class="social-media-links--platforms platforms vertical">
|
||||
<li>
|
||||
<a class="social-media-link-icon--youtube" href="https://www.youtube.com/user/ENISAvideos" target="_blank" >
|
||||
<span class='fab fa-youtube fa-2x'></span>
|
||||
<span class="platform-name">Youtube</span>
|
||||
</a>
|
||||
</li>
|
||||
<li>
|
||||
<a class="social-media-link-icon--twitter" href="https://x.com/enisa_eu" target="_blank" >
|
||||
<span class='fab fa-x-twitter fa-2x'></span>
|
||||
<span class="platform-name">X</span>
|
||||
</a>
|
||||
</li>
|
||||
<li>
|
||||
<a class="social-media-link-icon--linkedin" href="https://www.linkedin.com/company/european-union-agency-for-cybersecurity-enisa/" target="_blank" >
|
||||
<span class='fab fa-linkedin fa-2x'></span>
|
||||
<span class="platform-name">LinkedIn</span>
|
||||
</a>
|
||||
</li>
|
||||
<li>
|
||||
<a class="social-media-link-icon--facebook" href="https://www.facebook.com/ENISAEUAGENCY" target="_blank" >
|
||||
<span class='fab fa-facebook fa-2x'></span>
|
||||
<span class="platform-name">Facebook</span>
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-xs-12 col-md-6 col-lg-3 border-left">
|
||||
<nav aria-labelledby="block-enisaweb-contactus-menu" id="block-enisaweb-contactus" class="block block-menu navigation menu--contact-us">
|
||||
|
||||
<p id="block-enisaweb-contactus-menu">Contact us</p>
|
||||
|
||||
|
||||
|
||||
<ul data-block="footer" class="nav navbar-nav">
|
||||
<li class="nav-item">
|
||||
<a href="/about-enisa/contact/contact" class="nav-link" data-drupal-link-system-path="node/17344">Contacts</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/form/contact-form" class="nav-link" data-drupal-link-system-path="webform/contact_form">General queries</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/working-with-us/procurement" class="nav-link" data-drupal-link-system-path="node/17421">Public Procurement</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/form/media-inquiries" class="nav-link" data-drupal-link-system-path="webform/media_inquiries">Media inquiries</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
|
||||
|
||||
</nav>
|
||||
|
||||
|
||||
</div>
|
||||
<div class="col-xs-12 col-md-6 col-lg-3 border-left">
|
||||
<div id="block-enisaweb-findoutaboutus" class="block block-system block-system-menu-blockfind-out-about-us">
|
||||
|
||||
<p>Find out about us</p>
|
||||
|
||||
|
||||
<ul data-block="footer" class="nav navbar-nav">
|
||||
<li class="nav-item">
|
||||
<a href="/accessibility-statement" class="nav-link" data-drupal-link-system-path="node/18887">Accessibility</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/about-enisa/legal-notice" class="nav-link" data-drupal-link-system-path="node/17355">Legal Notice</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/about-enisa/data-protection/data-protection" class="nav-link" data-drupal-link-system-path="node/17346">Data Protection</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/about-enisa/cookies" class="nav-link" data-drupal-link-system-path="node/17345">Cookies</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/sitemap" target="_blank" class="nav-link" data-drupal-link-system-path="sitemap">Sitemap</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-xs-12 col-md-6 col-lg-3 border-left">
|
||||
<div id="block-enisaweb-pageofinterest" class="block block-system block-system-menu-blockpage-of-interest">
|
||||
|
||||
<p>Page of interest</p>
|
||||
|
||||
|
||||
<ul data-block="footer" class="nav navbar-nav">
|
||||
<li class="nav-item">
|
||||
<a href="/publications" class="nav-link" data-drupal-link-system-path="publications">Publications </a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/press-office" class="nav-link" data-drupal-link-system-path="node/11033">Press Office</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/digital-tools" class="nav-link" data-drupal-link-system-path="digital-tools">Digital Tools</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/work-with-us" class="nav-link" data-drupal-link-system-path="node/11443">Working with us</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a href="/about-enisa/public-access-to-documents" class="nav-link" data-drupal-link-system-path="node/17363">Public access to documents</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="line"></div>
|
||||
<div class="copy container">
|
||||
<div class="row">
|
||||
<div class="col-xs-12 col-md-12 col-lg-6 alignleft">
|
||||
<div id="block-enisaweb-copyrightfooter" class="block block-block-content block-block-content0f0b270e-cc5d-448c-a771-6cc7c0621340">
|
||||
|
||||
|
||||
|
||||
<div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>© 2026 by the European Union Agency for Cybersecurity</p>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
<div class="col-xs-12 col-md-12 col-lg-6 alignright">
|
||||
<div id="block-enisaweb-enisadescriptionfooter" class="block block-block-content block-block-content2b4d8729-4032-4438-ba87-1a58fbc364db">
|
||||
|
||||
|
||||
|
||||
<div class="clearfix text-formatted field field--name-body field--type-text-with-summary field--label-hidden field__item"><p>ENISA is an agency of the European Union</p>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p><a href="#" id="totop" class="totop"><span class="sr-only">Go to top</span></a></p>
|
||||
</footer>
|
||||
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
<script type="application/json">{"utility":"piwik","siteID":"5847bf6f-3ce3-4800-8749-1c565b34b7b6","sitePath":["www.enisa.europa.eu"],"instance":"ec"}</script>
|
||||
<script type="application/json">{"utility":"cck","url":"\/about-enisa\/cookies"}</script>
|
||||
|
||||
<script type="application/json" data-drupal-selector="drupal-settings-json">{"path":{"baseUrl":"\/","pathPrefix":"","currentPath":"taxonomy\/term\/1317","currentPathIsAdmin":false,"isFront":false,"currentLanguage":"en"},"pluralDelimiter":"\u0003","suppressDeprecationErrors":true,"ckeditorAccordion":{"accordionStyle":{"collapseAll":1,"keepRowsOpen":0,"animateAccordionOpenAndClose":1,"openTabsWithHash":0,"allowHtmlInTitles":1}},"csp":{"nonce":"wb2Cd5rPB6d4U2ahlwtoVw"},"user":{"uid":0,"permissionsHash":"c1c359b7541ecd1c4f0e321882d2e1eba1197d85c8ad3b45b15aff5871a9e6d0"}}</script>
|
||||
<script src="/sites/default/files/js/js_RQ57ED_QkadU0X-0Q8nEhDKVEkkdta8wY8_icwESnuY.js?scope=footer&delta=0&language=en&theme=enisaweb&include=eJxlj0EOgzAMBD-EG6kfQiYxEOrYNHGE8vuCWkql3mZn97IYgilKc_iB25hVrPMPCtE09-i95hBV3JfeE5LQkcSCGw3uBFgKMBoVu7qJdUCGYo2jTJdfnpVyA98805-tEUyrn2Gt4uefuqjcO6V-D6bKxYVcV-TbmaEkzMaKgXK3UZ9owkRS3ag57Ss4BBwGzhsvrCdlDA"></script>
|
||||
<script src="https://static.addtoany.com/menu/page.js" defer></script>
|
||||
<script src="/sites/default/files/js/js_kppnwVGSNMO58MOFQJXEYZNwpiIbQ8uG_I-yvuC5qBs.js?scope=footer&delta=2&language=en&theme=enisaweb&include=eJxlj0EOgzAMBD-EG6kfQiYxEOrYNHGE8vuCWkql3mZn97IYgilKc_iB25hVrPMPCtE09-i95hBV3JfeE5LQkcSCGw3uBFgKMBoVu7qJdUCGYo2jTJdfnpVyA98805-tEUyrn2Gt4uefuqjcO6V-D6bKxYVcV-TbmaEkzMaKgXK3UZ9owkRS3ag57Ss4BBwGzhsvrCdlDA"></script>
|
||||
<script src="/modules/contrib/ckeditor_accordion/js/accordion.frontend.min.js?telxj6"></script>
|
||||
<script src="/sites/default/files/js/js_fPzrD9aZOLJS9JI2GLgD7Zs-CzoWHT18p8hYIEuW9h4.js?scope=footer&delta=4&language=en&theme=enisaweb&include=eJxlj0EOgzAMBD-EG6kfQiYxEOrYNHGE8vuCWkql3mZn97IYgilKc_iB25hVrPMPCtE09-i95hBV3JfeE5LQkcSCGw3uBFgKMBoVu7qJdUCGYo2jTJdfnpVyA98805-tEUyrn2Gt4uefuqjcO6V-D6bKxYVcV-TbmaEkzMaKgXK3UZ9owkRS3ag57Ss4BBwGzhsvrCdlDA"></script>
|
||||
<script src="https://webtools.europa.eu/load.js" defer></script>
|
||||
<script src="/sites/default/files/js/js_Pj1gX-gXRHcdCBDI1-WO0jTP3o3GK7ZZPT2TZokpFjY.js?scope=footer&delta=6&language=en&theme=enisaweb&include=eJxlj0EOgzAMBD-EG6kfQiYxEOrYNHGE8vuCWkql3mZn97IYgilKc_iB25hVrPMPCtE09-i95hBV3JfeE5LQkcSCGw3uBFgKMBoVu7qJdUCGYo2jTJdfnpVyA98805-tEUyrn2Gt4uefuqjcO6V-D6bKxYVcV-TbmaEkzMaKgXK3UZ9owkRS3ag57Ss4BBwGzhsvrCdlDA"></script>
|
||||
|
||||
<script async="" src="/themes/custom/enisaweb/js/application.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user