The Geometry Behind Vector Search

Notes from linear algebra (lesson 1) and the embeddings lesson of my Master's course. Sources: Deisenroth et al., Mathematics for Machine Learning §6.4.6; 3Blue1Brown, Essence of Linear Algebrach. 1 and 9; Ethayarajh (2019); Timkey & van Schijndel (2021). Numbers below were re-run with NumPy 2.0.2 and PostgreSQL 18.6 + pgvector 0.8.7.

1. The dot product is a shadow

Dividing by $\|\mathbf{b}\|$ gives the signed length of the shadow of $\mathbf{a}$ on the line through $\mathbf{b}$. Multiplying that length by the unit vector $\mathbf{b}/\|\mathbf{b}\|$ gives the shadow itself. The $\|\mathbf{b}\|$ is divided out twice, which is why the denominator is $\mathbf{b}\cdot\mathbf{b}$:

With $\mathbf{a}=(3,4)$ and $\mathbf{b}=(4,3)$: the dot product is 24, $\cos\theta = 24/25 = 0.96$, the shadow has length 4.8 and equals $(3.84, 2.88)$. What is left, $\mathbf{a}-\operatorname{proj}_{\mathbf{b}}(\mathbf{a}) = (-0.84, 1.12)$, is perpendicular to $\mathbf{b}$. For $\mathbf{c}=(-4,3)$ the dot product with $\mathbf{a}$ is 0, so the shadow is empty.

Orthogonal means no component along the other vector, which is weaker than independent: $x=(-1,0,1)$ and $y=x^2=(1,0,1)$ have dot product 0 and correlation 0, yet $y$ is determined by $x$.

In $d$ dimensions the cosine of two independent random directions has a standard deviation of $1/\sqrt{d}$: 0.71 in 2D, 0.036 in 768D. So random vectors in high dimensions are nearly orthogonal.

2. Cosine and Euclidean agree on unit vectors

If $\|\mathbf{u}\|=\|\mathbf{v}\|=1$:

Distance is a decreasing function of cosine, so the smallest distance and the largest cosine pick the same neighbours. On random data (10,000 documents, 50 queries, 768 dimensions) the top-10 lists were identical for 50 of 50 queries after normalizing, and for 0 of 50 before. The difference shows up with raw counts: for $q=(1,0)$ and a document $d=(10,0)$ the Euclidean distance is 9 while the cosine is 1. An approximate index such as HNSW returns an approximation of this ranking.

3. What pgvector computes

• <=> cosine distance, $1-\cos(\mathbf{u},\mathbf{v})$
• <-> Euclidean (L2) distance
• <#> negative inner product, $-\,\mathbf{u}\cdot\mathbf{v}$

<#> is negative because Postgres only supports ascending index scans on operators (pgvector README), so the largest inner product has to be the smallest value. On unit vectors $-(\text{inner product})$ is the cosine; over 2,000 random unit vectors it matched 1 - (embedding <=> q) to 6e-8.

CREATE TABLE documents (
  id BIGSERIAL PRIMARY KEY,
  content TEXT NOT NULL,
  embedding vector(768) NOT NULL   -- L2-normalized at write time
);

CREATE INDEX ON documents USING hnsw (embedding vector_ip_ops);

SELECT id, content, (embedding <#> $1) * -1.0 AS cosine_similarity
FROM documents
ORDER BY embedding <#> $1 ASC
LIMIT 10;

The ASC query uses the HNSW index; with DESC the planner falls back to a sort over a sequential scan (a B-tree index can still scan backwards). On an exact scan of 20,000 rows, the three operators all took about 20 ms, so the simpler formula is not a speed argument.

4. A shared offset flattens cosine

For random variables with zero mean, covariance is an inner product: the norm is the standard deviation, cosine is the correlation, orthogonal means uncorrelated (MML §6.4.6). Cosine itself ignores the mean, so if all vectors share a large offset along one coordinate they all point the same way and every pair scores high. Ethayarajh and Timkey & van Schijndel describe this for contextual embeddings as anisotropy and rogue dimensions. Synthetic check, 2,000 independent vectors in 64 dimensions with one coordinate shifted and scaled:

Subtracting the mean restores the spread. In practice: subtract the corpus mean from the documents and the query, then normalize again. I only checked this on synthetic data, not on a real embedding model.