sharang/compliance-scanner-agent
1
0
Fork 0
Code Issues 79 Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

fix(llm): chunk embed() requests under the backend batch cap #221

Merged
sharang merged 1 commits from fix/embed-batch-chunking into main 2026-07-21 15:41:07 +00:00
Conversation 0 Commits 1 Files Changed 1
+49 -1
1 Commits
Author SHA1 Message Date
Sharang ParnerkarandClaude Fable 5 617b2df75e fix(llm): chunk embed() requests under the backend batch cap
CI / Check (push) Skipped
CI / Check (pull_request) Successful in 5m42s
Details
CI / Detect Changes (pull_request) Skipped
Details
CI / Deploy Agent (pull_request) Skipped
Details
CI / Deploy Dashboard (pull_request) Skipped
Details
CI / Deploy Docs (pull_request) Skipped
Details
CI / Deploy MCP (pull_request) Skipped
Details 
Found while prepping the C5 live test: the embeddings backend
(bge-multilingual-gemma2 via LiteLLM) caps input arrays at 25 per request
("given batch size overflow maximal one", max: 25), but embed() sent the whole
input in a single request. ControlIndex::build embeds the entire master-controls
corpus (~1.8k texts) in one embed() call, and the RAG pipeline batches docs too —
both 500 at scale. Unit tests passed only because they embed <=3 texts.

Fix: embed() now chunks into EMBED_BATCH_SIZE (16) requests and concatenates in
order; empty input short-circuits. Validated live — 1,784 texts over 112 chunked
requests succeed (~70s, one-time + cached by the index).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-21 17:34:17 +02:00