Embeddings and RAG: Design Retrieval Before the Prompt

1. Start with the questions you need to answer#

A RAG system has at least two separate jobs: finding relevant evidence and building an answer from it. Evaluate them separately. If search returns the wrong document, switching to a different generative model may only hide the failure behind a more convincing explanation.

Write down questions together with the documents or chunks that correctly answer them. Include exact lookups by identifier, paraphrases, internal jargon, and questions that have no answer in the corpus. In a code repository, a function signature and a question about behavior may need different retrieval paths. Test lexical search alongside semantic search before assuming that everything needs vectors.

2. Chunk while preserving structure and provenance#

A chunk needs enough context to be understood on its own and enough focus that it does not dilute the topic. Keep headings, file paths, document version, and function boundaries where they make sense. Record a stable identifier and a content hash so you can detect changes. Do not split tables or code samples in half unless you have a strategy for reassembling them.

There is no universal chunk size. Compare several configurations against the same questions and review the failures. A large overlap can improve continuity, but it also duplicates retrieved text and increases ingestion and storage costs. Measure how many chunks contribute distinct evidence to the final context.

• Store document_id, chunk_id, version, provenance, and permissions alongside the vector.
• Detect deletions and access changes; updating text is not the only sync operation you need.
• Use the tokenizer and limits that apply to your provider. Characters are not tokens.

3. Version the vector space#

Dimensionality defines the shape of a vector, not its meaning. Two models that both return 1024 numbers do not necessarily produce compatible spaces. Version the model, dimensionality, normalization, and chunking strategy as a single unit. Changing any of them requires a new evaluation and, whenever the space changes, a separate index.

Some providers distinguish a query input type from a document input type. That setting is part of the contract, so do not leave it out when comparing results. During ingestion, it is also better to fail explicitly on inputs that are too long than to truncate them silently and lose the paragraph that held the answer.

Documentation: Voyage: input types, dimensions, and truncation ↗ · Mistral: code embeddings ↗

4. Authorize before you retrieve#

The tenant or workspace used for search must come from a session verified on the server. An identifier sent by the browser does not prove access. Apply visibility restrictions inside the index query itself, and check permissions again when loading content. Filtering after chunks have already been handed to the model is too late.

session = requireVerifiedSession(request)
scope = authorizeWorkspace(session, requestedWorkspace)
queryVector = embedQuery(query, indexVersion)
hits = searchIndex(queryVector, {
  workspace: scope.id,
  allowedResources: scope.resourceIds,
  indexVersion
})
context = loadAuthorizedChunks(session, hits)
answer = generateWithCitations(query, context)

A shared cache must include the authorization scope and the index version in its key. Design invalidation for permission revocations as well. Isolation does not end at the vector query: it extends to citations, download links, traces, and stored answers.

5. Measure retrieval before generation#

For questions with several known relevant chunks, recall@k is the fraction of them that appear in the top k results. If you only mark one acceptable document per question, you can also measure the share of questions where it shows up in that top k. Define your unit of relevance precisely; switching from documents to chunks changes how the number should be read.

The next step is evaluating the answer: grounding in sources, contradictions, citations that actually contain the claim, and the ability to abstain. Also log cases where no evidence was found. A system that always answers can look excellent on the surface while being unreliable in practice.

6. Migrate with a side-by-side comparison#

Build the new index next to the old one and replay an approved set of queries against both. Check quality, permissions, latency, and cost before switching reads over. Keep an ingestion cursor so you can pick up changes that happen during the migration. The cutover must include a clear rollback point.

Budget for the initial ingestion, ongoing updates, stored vectors, the search index, and the context sent to the generator. The embedding price covers only part of the bill. Use the comparison tool to shortlist candidates, then confirm the result on your own corpus, especially if it contains code, images, or multilingual documents.

Documentation: Cohere: Embed modalities and dimensions ↗