fix(llm): chunk embed() requests under the backend batch cap #221

2026-07-21T15:34:18Z

sharang commented

2026-07-21 15:34:18 +00:00

Bug found while prepping the C5 live test. The embeddings backend (bge-multilingual-gemma2 via LiteLLM) caps input arrays at 25 per request ("given batch size overflow maximal one", max: 25), but embed() sent the whole input in one request.

Impact: ControlIndex::build embeds the entire master-controls corpus (~1,784 code-checkable texts) in a single embed() call → HTTP 500. The RAG pipeline batches docs the same way → same latent bug. Unit tests passed only because they embed ≤3 texts.

Fix: embed() chunks into EMBED_BATCH_SIZE (16) requests and concatenates results in input order; empty input short-circuits (no HTTP).

Validated live: 1,784 texts over 112 chunked requests succeed (~70s, one-time — the control index caches it). Green locally (clippy -D warnings, fmt, full suite; +2 embedding tests).

Unblocks C5 (semantic master-controls mapping) at real corpus scale.

🤖 Generated with Claude Code

**Bug found while prepping the C5 live test.** The embeddings backend (bge-multilingual-gemma2 via LiteLLM) caps input arrays at **25 per request** (`"given batch size overflow maximal one", max: 25`), but `embed()` sent the whole input in one request. Impact: `ControlIndex::build` embeds the entire master-controls corpus (~1,784 code-checkable texts) in a single `embed()` call → HTTP 500. The RAG pipeline batches docs the same way → same latent bug. Unit tests passed only because they embed ≤3 texts. **Fix:** `embed()` chunks into `EMBED_BATCH_SIZE` (16) requests and concatenates results in input order; empty input short-circuits (no HTTP). **Validated live:** 1,784 texts over 112 chunked requests succeed (~70s, one-time — the control index caches it). Green locally (clippy -D warnings, fmt, full suite; +2 embedding tests). Unblocks C5 (semantic master-controls mapping) at real corpus scale. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

sharang added 1 commit 2026-07-21 15:34:19 +00:00

fix(llm): chunk embed() requests under the backend batch cap

CI / Check (push) Skipped

CI / Check (pull_request) Successful in 5m42s

Details

CI / Detect Changes (pull_request) Skipped

Details

CI / Deploy Agent (pull_request) Skipped

Details

CI / Deploy Dashboard (pull_request) Skipped

Details

CI / Deploy Docs (pull_request) Skipped

Details

CI / Deploy MCP (pull_request) Skipped

Details

617b2df75e

Found while prepping the C5 live test: the embeddings backend
(bge-multilingual-gemma2 via LiteLLM) caps input arrays at 25 per request
("given batch size overflow maximal one", max: 25), but embed() sent the whole
input in a single request. ControlIndex::build embeds the entire master-controls
corpus (~1.8k texts) in one embed() call, and the RAG pipeline batches docs too —
both 500 at scale. Unit tests passed only because they embed <=3 texts.

Fix: embed() now chunks into EMBED_BATCH_SIZE (16) requests and concatenates in
order; empty input short-circuits. Validated live — 1,784 texts over 112 chunked
requests succeed (~70s, one-time + cached by the index).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

sharang merged commit 60601d8215 into main

2026-07-21 15:41:07 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sharang/compliance-scanner-agent#221