Follow bytes through a write
An index lookup, a durable commit and a replicated read are separate operations. Draw where bytes reside: process memory, filesystem cache, device and replica. Acknowledgment policy tells the client which failures an accepted write should survive.
A simplified LSM write path is log append, memory-table update, flush to sorted files and later compaction. A page-oriented engine changes buffered pages and relies on its recovery protocol before dirty pages reach storage. Exact ordering, log format and durability settings depend on the engine.
B-tree pages and LSM sorted runs
B-tree-family indexes keep a bounded branching hierarchy and sorted leaf access. Inserting a key can split a page; random insertion, page size, fill factor and buffering affect cost. Sequential-looking IDs may improve locality while concentrating concurrent work on a hot leaf.
LSM sorted files make foreground writes efficient, but reads may consult several runs. Compaction merges files and removes obsolete entries where safe. A storage choice is a workload decision, not a rule that one structure always wins.
Three amplification budgets
| Amplification | Compare | Consequence |
|---|---|---|
| Write | Bytes written below the engine vs logical updates | Device wear and sustained I/O |
| Read | Work needed to answer a logical read | Latency and cache pressure |
| Space | Physical retained bytes vs live logical bytes | Capacity and recovery headroom |
Leveled and tiered compaction choose different compromises. More overlap can reduce rewrite work but make reads inspect more files. Compaction cannot catch up if its input rate exceeds its available resources; stalls are a backpressure mechanism. Allocate spare disk for compaction output and temporary copies rather than sizing only the final dataset.
MVCC and long-running readers
Multi-version concurrency control lets readers observe an appropriate snapshot while writes create new versions. Old versions cannot be reclaimed while still needed by active snapshots. A long transaction can therefore retain garbage and expand storage even if write throughput is modest.
Track transaction age, dead versions, cleanup progress and lock waits. A logical delete is not instantaneous physical erasure. Large backfills and exports must avoid holding unnecessary snapshots for hours. Isolation semantics still need explicit analysis; MVCC alone does not imply serializability.
Read repair, tombstones and compaction
A deleted value may leave a tombstone so replicas know it was removed. Repair and retention determine when that marker may be discarded. Removing the marker while a stale replica still holds the value can resurrect it.
A Bloom filter can reduce unnecessary file probes, but cannot replace the exact index or safely decide uniqueness. Cache index blocks and data blocks according to workload, and measure cold-read behavior after restart. A benchmark with a fully warm working set does not predict disaster-recovery latency.
Full-text retrieval pipeline
Document -> language analysis -> terms/positions -> inverted index
Query -> compatible analysis -> candidates -> ranking -> permission check -> results
Tokenization affects identifiers, hyphens, case, synonyms and languages. Phrase queries need position information. Ranking such as BM25 uses term and document statistics; tune it with a relevance set rather than assuming a scoring formula understands user intent.
Permissions must constrain the candidate/result set before private content can be returned, highlighted or summarized. Filtering only a displayed title can still leak snippets or counts. Search-index updates and permission revocation need an explicit freshness objective.
Autocomplete and pagination
Prefix indexes, tries and completion structures serve different matching needs. A course title completion can use a curated public vocabulary; raw private query history should not become shared suggestions. Apply abuse limits because prefix expansion and wildcard queries can be expensive.
Stable search pagination needs an ordering tie-breaker and a consistent view when results change. A cursor might encode score, document ID and snapshot identity. Large offsets repeat work and may skip or duplicate results as ranking changes. Expire snapshots so abandoned sessions cannot pin storage indefinitely.
Vector and hybrid search
An embedding maps content into a numerical space. Similarity is not truth, identity or permission. Exact nearest-neighbor search compares the eligible vectors; approximate structures such as HNSW trade retrieval recall for speed and memory.
Filtering matters: searching a broad index and discarding unauthorized candidates afterwards can return too few valid results. Choose a strategy that searches the permitted subset or expands candidates appropriately, then verify authorization before returning any content. Measure recall and latency on representative filters.
Hybrid retrieval combines lexical and vector candidates, optionally fusing rank positions before reranking. Lexical retrieval helps exact error codes; vectors help paraphrases. Evaluate both against actual question-document pairs. More retrieved text can dilute relevance and increase cost.
Version the embedding model, dimensions, chunking and normalization. A new model can require re-embedding; vectors from unrelated spaces should not be casually compared. Rebuild into a new index, measure quality, then switch readers. Deletion must remove chunks, source records and cached retrieval results.
Object durability and repair
Replication stores complete copies. Erasure coding stores fragments with redundancy, allowing reconstruction within the scheme's failure assumptions. It saves some storage but changes reconstruction traffic, CPU and failure behavior. Choose fault-domain placement so one correlated outage does not remove too many fragments.
Checksums help detect corruption, not repair it alone. Scrubbing verifies data over time; repair needs healthy copies/fragments and available capacity. Track repair backlog and restore throughput as well as durability objectives. Small metadata records can remain the critical availability dependency even when large objects survive.
Worked capacity estimate
A hypothetical 50 million 768-dimensional vectors require about 153.6 GB for raw float32 components alone. IDs, graph links, filters, replicas and operational headroom add to this. Float16 halves raw component bytes but changes precision and does not halve every index overhead.
Start with measured recall/latency and memory at a representative scale. Partition by tenant only when it matches query and isolation requirements; thousands of tiny indexes can introduce different costs from one shared filtered index.
Exercise
Design search over public course notes and private student documents. Separate their trust boundaries, define deletion propagation and show an index rebuild while writes continue. Compare lexical-only, vector-only and hybrid results using a labelled evaluation set. Continue to data lifecycle.