fix(rag): strip HTML tags before chunking + D5 re-ingestion scripts
HTML files from gesetze-im-internet.de were decoded as raw UTF-8, keeping <div>/<p> tags intact. The legal chunker regex requires § at line start, which never matched inside HTML tags → 0% section metadata for HTML docs. Fix: detect HTML content and strip tags before sending to embedding service. Block elements become newlines, entities are decoded. § signs now appear at line starts → section detection works. Also adds D5 re-ingestion scripts (reingest_d5.py + config) for batch re-processing of all documents in Qdrant collections. 27 rag-service tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -7,6 +7,7 @@ from pydantic import BaseModel
|
||||
|
||||
from api.auth import optional_jwt_auth
|
||||
from embedding_client import embedding_client
|
||||
from html_utils import looks_like_html, strip_html
|
||||
from minio_client_wrapper import minio_wrapper
|
||||
from qdrant_client_wrapper import qdrant_wrapper
|
||||
|
||||
@@ -111,6 +112,11 @@ async def upload_document(
|
||||
if not text or not text.strip():
|
||||
raise HTTPException(status_code=400, detail="Could not extract any text from the document")
|
||||
|
||||
# --- Strip HTML if detected ---
|
||||
if looks_like_html(text):
|
||||
text = strip_html(text)
|
||||
logger.info("Stripped HTML tags from %s", filename)
|
||||
|
||||
# --- Chunk ---
|
||||
try:
|
||||
chunk_result = await embedding_client.chunk_text(
|
||||
|
||||
Reference in New Issue
Block a user