Vector Database Illustrated
An illustrated walkthrough: index three sentences, run a query, and find the nearest neighbour — every cell is arithmetic you can follow on screen.
Vector databases are the backbone of Retrieval Augmented Generation (RAG). How do they actually work?
An LLM does not “look up” your docs the way a search engine indexes keywords. Text is turned into vectors — lists of numbers — so that similar meaning sits nearby in space. A vector database stores those numbers and, at query time, finds the closest ones.
Goal for this page: index three tiny sentences, push a query through the same pipeline, and recover the nearest neighbour step by step. In production the corpus is millions of chunks; here every matrix fits on one screen.
Start with three sentences to store, and one query to search.
Data is what you index once (documents, FAQ rows, code comments). Query is what the user asks later. Both must travel through the same embedding pipeline so they land in the same geometric space — otherwise “nearby” would be meaningless.
In practice: millions of chunks, thousands of tokens each (e.g. 32k context windows). The steps do not change — only the scale.
Each word becomes a 4-number vector from the lookup table.
Computers do not understand the string who.
An embedding table maps every vocabulary word to a fixed list of numbers.
Words used in similar contexts tend to get similar vectors — that is the whole trick.
Highlighted columns are the words we actually need. Copy each column under its sentence: three words → a 4×3 matrix.
A linear layer + ReLU turns word vectors into feature vectors.
Raw word embeddings are a static dictionary. The encoder mixes the dimensions
(linear: multiply by weights, add bias) then zeros out negatives
(ReLU: h = max(0, z)).
That is a baby version of what a transformer block does with attention + MLPs.
With input embedding [0, 1, 0, 0] and the weight matrix below (last column = bias):
z = [1, 1, −1, −1] → ReLU → [1, 1, 0, 0]
Negatives become zero. Repeat for every word column to fill the green matrices.
Each green matrix still has one column per word. Next we squash the sequence into a single vector.
Three word columns collapse into one sentence vector.
Average across the three columns, row by row. The result is what people call a text embedding or sentence embedding — one vector that stands for the whole phrase. (Other options exist, like taking a special CLS token; mean pooling is the most common default.)
mean([0,1,1,1], [0,2,0,0], [1,1,0,0]) = [1/3, 4/3, 1/3, 1/3]
Project 4D → 2D and store the short vectors.
Comparing full embedding vectors at billion scale is expensive in memory and CPU. A projection (here a simple 2×4 matrix) hashes each sentence into a shorter index vector. Those short vectors are what live in vector storage.
For “how are you”: index = [e₁+e₂, e₃+e₄] = [5/3, 2/3].
Repeat for every sentence. The query uses the identical projection so it lands on the same map.
Same pipeline: lookup → encode → pool → project. The database is now indexed; the query is a point in the same 2D space.
Score every stored vector with a dot product — pick the max.
The dot product q · v measures how aligned the query is with each stored vector. Larger means more similar (for this setup). Doing it as a matrix multiply — transpose the query and multiply once — scores the whole database in one shot.
how are you · q = (5/3)(8/3) + (2/3)(2/3) = 40/9 + 4/9 = 44/9
who are you · q = (5/3)(8/3) + (0)(2/3) = 40/9
who am I · q = (7/3)(8/3) + (2/3)(2/3) = 56/9 + 4/9 = 60/9 ← max
60/9 beats 44/9 and 40/9. Linear scan is fine for three vectors; at billions of rows it is the bottleneck — production systems use approximate nearest neighbour indexes such as HNSW (graph search) or IVF (cluster then scan a few buckets).
This toy page
3 vectors · exact scan · dot product · 2D indexes
Every cell is doable with pen and paper.
Production vector DB
Millions–billions of vectors · ANN (HNSW / IVF / PQ) · often cosine on normalized embeddings · 384–3072 dims
Same ideas: embed, store, score, return top-k.
Where RAG uses this
User question → embed → retrieve top chunks → stuff into the LLM prompt as context → generate the answer.
Retrieval quality often matters more than the LLM choice.
Takeaway
A vector database is an embedding pipeline, a projection into a compact index, and a similarity search (here: dot product + argmax). Every step on this page is arithmetic you can do in pen — worth remembering when the word “database” makes it sound like something mystical.
tokens → word vectors → encode → mean pool → project → store; query repeats the same path → score against storage → return nearest text.