Benjamin Admin 0b0eed27b0 feat(embedding): NIST PDF text normalization + safe re-ingest script ...

Fix broken multi-column PDF extraction for NIST/BSI/ENISA documents:
- _normalize_pdf_text(): fixes broken section numbers (1 . 1 → 1.1),
  control IDs (AC - 1 → AC-1), ligatures, soft hyphens
- pdfplumber tolerances increased (x=3,y=4) for better column handling
- 3 new regex patterns: NIST CSF 2.0, NIST enhancements, OWASP Top 10
- reingest_nist.py: safe upload-before-delete for 4 lost NIST PDFs
- reingest_d5.py: safety fix — upload first, verify, then delete old

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-05-03 06:42:46 +02:00

..

import_backup.py

…

ingest_bag_urteile.py

…

migrate_jsonb.py

…

quality_report.py

docs: session handover — D2-D5 complete, quality report, NIST plan

2026-05-02 22:22:55 +02:00

reingest_d5_config.py

fix(rag): strip HTML tags before chunking + D5 re-ingestion scripts

2026-05-02 08:18:25 +02:00

reingest_d5.py

feat(embedding): NIST PDF text normalization + safe re-ingest script

2026-05-03 06:42:46 +02:00

reingest_nist.py

feat(embedding): NIST PDF text normalization + safe re-ingest script

2026-05-03 06:42:46 +02:00

replace_eu_pdfs_with_html.py

feat(embedding): add NIST/ENISA/standard section numbering to chunker

2026-05-02 19:24:10 +02:00

test_d4_integration.py

feat(pipeline): structural metadata end-to-end (Blocks D2-D4)

2026-05-01 20:34:00 +02:00

test_qwen_backfill.py

perf(pipeline): switch to v3 prompt for generation, v4 fields via Haiku backfill

2026-04-27 00:14:47 +02:00