Embeddings, Vector Databases and Semantic Search
What embeddings are, similarity metrics, ANN indexes (HNSW, IVF), quantization, choosing a vector store, hybrid search and operating vector indexes at scale.
Embeddings
An embedding model maps text (or images/audio) to a dense vector (e.g. 384–3,072 dimensions) such that semantically similar items are close. Similarity is measured with cosine similarity, dot product or Euclidean distance (use the metric the model was trained for).
Approximate nearest neighbour (ANN) indexes
Exact search compares the query with every vector (O(n)); ANN indexes trade a little recall for speed.
| Index | How | Notes |
|---|---|---|
| HNSW | Multi-layer proximity graph | High recall, fast queries, more memory; tune M, ef_construction, ef_search |
| IVF (inverted file) | Cluster vectors; search nearest clusters | Less memory; tune number of lists/probes |
| PQ / scalar / binary quantization | Compress vectors | 4–32× memory reduction; rescore top results with full vectors |
| DiskANN-style | Graph on SSD | Very large datasets at lower memory |
flowchart LR T[Text chunk] --> EM[Embedding model] --> V[Vector 1024-d] V --> IDX[(HNSW index + metadata)] Q[Query text] --> EM2[Same embedding model] --> QV[Query vector] QV --> IDX --> TOP[Top-k similar chunks]
Choosing a vector store
| Option | Fits when |
|---|---|
| pgvector (PostgreSQL) | Up to millions of vectors; want SQL joins, transactions and permissions in one DB |
| OpenSearch / Elasticsearch / Azure AI Search | Hybrid keyword + vector search, filters, existing search platform |
| Managed vector services (Vertex AI Vector Search, Pinecone) | Very large scale, low ops |
| Qdrant, Weaviate, Milvus | Dedicated vector features, self-hosted or cloud |
| Cloud object-storage vectors (e.g. S3 Vectors) | Large, cost-sensitive, less latency-critical collections |
Operational concerns
- Embedding model versioning: vectors from different models are incompatible. Store the model version per vector; re-embed into a new index and switch with an alias.
- Metadata filtering: pre-filtering vs post-filtering affects recall; prefer engines with efficient filtered ANN.
- Updates and deletes: HNSW handles deletes with tombstones; plan compaction/reindexing.
- Multi-tenancy: index per tenant or tenant filter; ensure isolation.
Semantic vs keyword search
Semantic search handles synonyms and paraphrases; keyword search handles exact identifiers, codes and rare terms. Hybrid search combines both and is the default recommendation.
Interview questions and answers
How much memory do 100 million 768-dimension float32 vectors need?
100M × 768 × 4 bytes ≈ 307 GB before index overhead (HNSW adds graph links). Scalar int8 quantization reduces it to ~77 GB; binary quantization to ~10 GB, with rescoring to recover accuracy.
How do you change the embedding model without downtime?
Build a new index with the new model in parallel (backfill), evaluate retrieval quality, dual-write new content to both, switch the read alias, then delete the old index.