Commit Graph

4 Commits

Author SHA1 Message Date
Benjamin Admin
c52dbdb8f1 feat(rag): optimize RAG pipeline — JSON-Mode, CoT, Hybrid Search, Re-Ranking, Cross-Reg Dedup, chunk 1024
Some checks failed
CI/CD / go-lint (push) Has been skipped
CI/CD / python-lint (push) Has been skipped
CI/CD / nodejs-lint (push) Has been skipped
CI/CD / test-go-ai-compliance (push) Failing after 42s
CI/CD / test-python-backend-compliance (push) Successful in 1m38s
CI/CD / test-python-document-crawler (push) Successful in 20s
CI/CD / test-python-dsms-gateway (push) Successful in 17s
CI/CD / validate-canonical-controls (push) Successful in 10s
CI/CD / Deploy (push) Has been skipped
Phase 1 (LLM Quality):
- Add format=json to all Ollama payloads (obligation_extractor, control_generator, citation_backfill)
- Add Chain-of-Thought analysis steps to Pass 0a/0b system prompts

Phase 2 (Retrieval Quality):
- Hybrid search via Qdrant Query API with RRF fusion + automatic text index (legal_rag.go)
- Fallback to dense-only search if Query API unavailable
- Cross-encoder re-ranking with BGE Reranker v2 (RERANK_ENABLED=false by default)
- CPU-only PyTorch dependency to keep Docker image small

Phase 3 (Data Layer):
- Cross-regulation dedup pass (threshold 0.95) links controls across regulations
- DedupResult.link_type field distinguishes dedup_merge vs cross_regulation
- Chunk size defaults updated 512/50 → 1024/128 for new ingestions only
- Existing collections and controls are NOT affected

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-21 11:49:43 +01:00
Benjamin Admin
ee79a48e8e fix: add User-Agent header to PDF downloads
Some sites (ENISA) reject requests without User-Agent header.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 09:33:06 +01:00
Benjamin Admin
9026e392dc fix: ENISA URLs + increase curl timeout for large PDFs
- Update ENISA download URLs to new site structure (publications → sites/default/files)
- Increase curl max-time from 300s to 600s for IFRS PDFs (7.5-8.2MB)
- Update ENISA Secure by Design metadata (title changed)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 09:07:01 +01:00
Benjamin Admin
9496e758fc feat: EU-IFRS 2023/1803 + EFRAG Endorsement ingestion & system prompt
- Ingestion script: Add 3 new PDFs (IFRS DE/EN, EFRAG Endorsement Status)
  to ingest-industry-compliance.sh (7 → 10 documents total)
- System prompt: Add EU-IFRS and EFRAG to competence area, add mandatory
  IFRS endorsement warning section for all IFRS/IAS queries

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 01:56:04 +01:00