This is the failure mode nobody warns you about. I run a RAG assistant on pgvector, and the quietest breakages I've seen came from index changes, not prompt changes: a chunking tweak, a re-embedded batch, and suddenly retrieval quality shifts with no error anywhere. What saved me was treating the index like a migration. Every chunking or embedding change gets versioned, and a retrieval eval runs against a fixed question set before and after. If recall moves, the change doesn't ship. The schema discipline framing is exactly right: an unversioned index is a silent liar.
Love the "treat the index like a migration" framing, Akanksha. A fixed question set run before and after is exactly the gate we push for too. Curious: do you version the chunker config and the embedding model together as one release, or separately?
Great question, Daniel! I prefer tracking the chunking config and embedding model separately, since either can change independently. But I treat the resulting index as a single versioned release, with both configurations recorded together. That way, every retrieval evaluation maps to a specific index version, and it's easier to identify what caused a regression.
That's a clean split, Akanksha: track the inputs separately, but ship one index version so every eval result points back to a single thing. When a regression does show up, do you bisect it by rebuilding with one config rolled back, or does the recorded metadata usually make the culprit obvious?
Usually the metadata gets me most of the way there. Every eval run records which chunker config and embedding version produced the index, so a regression points at the diff between the last green run and the current one. When it's ambiguous, usually when both changed at once, I rebuild with one rolled back. Slower, but definitive. The discipline of never changing both without a tagged release in between is what keeps the ambiguous cases rare.
This is the failure mode nobody warns you about. I run a RAG assistant on pgvector, and the quietest breakages I've seen came from index changes, not prompt changes: a chunking tweak, a re-embedded batch, and suddenly retrieval quality shifts with no error anywhere. What saved me was treating the index like a migration. Every chunking or embedding change gets versioned, and a retrieval eval runs against a fixed question set before and after. If recall moves, the change doesn't ship. The schema discipline framing is exactly right: an unversioned index is a silent liar.
Love the "treat the index like a migration" framing, Akanksha. A fixed question set run before and after is exactly the gate we push for too. Curious: do you version the chunker config and the embedding model together as one release, or separately?
Great question, Daniel! I prefer tracking the chunking config and embedding model separately, since either can change independently. But I treat the resulting index as a single versioned release, with both configurations recorded together. That way, every retrieval evaluation maps to a specific index version, and it's easier to identify what caused a regression.
That's a clean split, Akanksha: track the inputs separately, but ship one index version so every eval result points back to a single thing. When a regression does show up, do you bisect it by rebuilding with one config rolled back, or does the recorded metadata usually make the culprit obvious?
Usually the metadata gets me most of the way there. Every eval run records which chunker config and embedding version produced the index, so a regression points at the diff between the last green run and the current one. When it's ambiguous, usually when both changed at once, I rebuild with one rolled back. Slower, but definitive. The discipline of never changing both without a tagged release in between is what keeps the ambiguous cases rare.