Embeddings, vector search and RAG#

Embeddings turn text, images or other content into lists of numbers that place similar meanings close together; vector search finds the closest ones quickly; retrieval-augmented generation (RAG) uses those results to give a language model the right documents to answer from.

Before reading

Unstructured and multimodal data

Why it matters#

Much of a company's knowledge is in documents, not tables: manuals, support articles, policies, contracts, tickets. Keyword search misses documents that use different words for the same thing. Language models answer fluently but know nothing about internal documents and invent answers when unsure. Embeddings and retrieval connect the two, and make unstructured content one more governed dataset on the platform.

How it works#

An embedding model reads a piece of content and returns a vector, a list of a few hundred to a few thousand numbers. Content with similar meaning gets vectors that are close together, measured by cosine similarity or a similar distance. "How do I return a damaged item?" and "refund for broken product" end up near each other even though they share no words. The same model must be used for the documents and for the questions; vectors from different models cannot be compared.

Documents are first split into chunks, such as a few paragraphs each, because a whole manual is too long to embed meaningfully or to give to a model. Each chunk is stored with its vector, its text, and metadata: source document, section, date, product, language and access permissions.

Vector search finds the chunks nearest to a question's vector. Comparing against every chunk is exact but slow for millions of chunks, so vector indexes use approximate nearest neighbour (ANN) methods such as HNSW, a navigable graph of neighbours, or IVF, which searches only the closest clusters. They trade a little accuracy for large speed gains. Vector search is offered by dedicated vector databases, by extensions to existing databases such as pgvector for PostgreSQL, by search engines, and increasingly by lakehouse and warehouse platforms. Combining vector search with keyword search, called hybrid search, usually finds more relevant results than either alone, especially for product codes and names.

Retrieval-augmented generation puts the pieces together. The user's question is embedded; the most relevant chunks are retrieved, filtered by metadata and permissions; they are placed in the model's prompt with an instruction to answer only from them and cite them; the model answers. Answers are grounded in current documents rather than in what the model memorised, and can link to their sources.

flowchart LR
  subgraph Index[Indexing, when documents change]
    direction TB
    D[Manuals and<br/>support articles] --> C[Split into chunks,<br/>add metadata and<br/>permissions] --> E[Embed] --> V[(Vector index)]
  end
  Q[Question] --> QE[Embed question]
  QE --> S[Search: nearest chunks<br/>the user may read]
  V --> S
  S --> L[Model answers from<br/>retrieved chunks,<br/>with citations]

Access rules must follow the documents. If a chunk comes from a document only the finance team may read, it must be filtered out of results for everyone else before it reaches the model; a model given a chunk will use it. Permissions are stored as metadata on each chunk and applied as a filter at search time, using the user's identity. When document permissions change, chunk metadata must change too.

The index is derived data, managed like any other: rebuilt when documents change, versioned by embedding model, monitored for freshness, and subject to the same retention and erasure rules as the documents. Deleting a document must delete its chunks.

Key terms#

Embedding A vector representing the meaning of a piece of content.
Chunk A section of a document, embedded and retrieved as a unit.
Cosine similarity A measure of how close two vectors point in the same direction.
Approximate nearest neighbour (ANN) Fast search that finds close vectors with high, not perfect, accuracy.
Hybrid search Combining vector similarity with keyword matching.
Retrieval-augmented generation (RAG) Answering with a model that is given retrieved documents.

Example#

The shop's support assistant answers customer service staff from product manuals and support articles.

Indexing: each article and manual is split into chunks of about 300 words, embedded, and stored with product_id, language, updated_at and audience (public, staff or finance).
Question: a staff member asks "can the X200 kettle be descaled with vinegar?".
Retrieval: hybrid search finds chunks mentioning the X200 and descaling; the filter keeps only chunks with audience public or staff.
Answer: the model replies that the manual recommends citric acid and warns against vinegar for the X200, citing manual section 6.2 and a support article.

When the manual is updated, its chunks are re-embedded overnight; when an article is withdrawn, its chunks are deleted from the index.

Pitfalls#

No permission filter. Retrieval leaks restricted documents through the model's answers.
Mixing embedding models. Vectors from different models are not comparable; re-embed everything when the model changes.
Chunks too large or too small. Large chunks dilute relevance; tiny chunks lose context.
Vector search alone for codes and names. Exact identifiers are found better by keyword search.
Stale indexes. Answers quote documents that have been changed or withdrawn.
Filtering after the ANN search. Restrictive permission filters return too few results, or none; use an index that filters during search.

Learn more#

Agents and MCP Row-level security and masking LLM applications and evaluation Fine-tuning or RAG