19-07-2026 Release v0.12

Search moves into the file

For most of the last decade, “add search to your app” has meant adding a system. You keep your rows in Postgres or SQLite, then you stand up Elasticsearch or an FTS5 table for text, and — once the AI wave arrived — a third box for vectors: pgvector, Qdrant, Pinecone. Three engines, three deployments, three things to back up. And in the seams between them, the same data, copied again and again, drifting out of sync the moment a write lands in one place but not yet the others.

In v0.12, full-text and vector search stop being a second and third system. They live inside the same encrypted, versioned Arkeion file, next to your SQL — over one hash-chained history, reachable with the same AS OF, provable with the same verify().

Three systems and the plumbing between them, versus one file that already holds all three.

The second-system tax

The cost of a separate search stack is rarely the search engine itself — it’s everything around it. Every document you want searchable has to be shipped to the other system: an indexing pipeline, a queue, a reconciliation job for when the pipeline drops a message. Your data now lives in two or three places, which means two or three copies to secure, and two or three answers to the question “is this in sync right now?” The honest answer, under load, is usually not quite.

Putting search in the file collapses all of that. When you INSERT a row, its text is indexed in the same transaction; when you write an embedding, it joins the same index atomically. There is no window where the row exists but the search result doesn’t, because there is no second system to fall behind.

SQL, full-text and vectors are three surfaces over one file — and the whole file is versioned, provable and time-travelling.

Full-text: how BM25 lives in the file

Full-text search is, underneath, an inverted index. Where a normal index maps a row to its values, an inverted index maps each term to the list of documents that contain it. Text comes in, gets tokenised into terms, and each term points at a posting list: the documents it appears in, with positions for phrase and snippet queries.

An inverted index: terms in a sorted dictionary, each pointing at the documents that contain it. Sorting the terms lets the postings share prefixes and compress.

Two design choices make Arkeion’s version compact. The term dictionary is sorted, so adjacent terms share long prefixes and store only the difference; and the posting lists are prefix-compressed rather than written out in full. The result, measured on the same corpus, is an index that came out smaller than SQLite’s FTS5 — while ranking with BM25 (the standard relevance model that balances term frequency against how common a term is) and returning highlighted snippets.

We’ll be honest about the trade: FTS5 still builds and queries faster. It has had years of tuning and it isn’t paying for versioning. What Arkeion buys with the difference is something FTS5 structurally can’t offer — search that remembers.

Search that time-travels

Because every posting lives on the same hash-chained history as your rows, the index has a past. MATCH … AS OF runs a full-text query against the file as it stood at an earlier version — not a backup, not a snapshot job, just the index itself, addressable in time.

The index carries its own history: a full-text query can be aimed at any past version of the file.

Vector: nearest-neighbour without the third box

Vector search answers a different question — not “which documents contain this word” but “which embeddings are closest to this one.” Done naively that means comparing the query against every vector, which is linear and slow. Arkeion uses IVF (inverted file): during the build, the vector space is partitioned into cells around representative centroids. At query time you only probe the handful of cells nearest the query, not the whole set.

IVF turns a full scan into a few cell probes; product quantization then compresses each vector, and an int8 re-rank pass restores the precision PQ trades away.

On top of IVF, product quantization (the PQ in IVF-PQ) compresses each vector by replacing it with codes from small learned codebooks — the reason the index is so small. Because PQ is lossy, a final int8 re-rank pass rescreens the top candidates at higher precision, recovering the accuracy quantization gives up. And when you want the best of both worlds, hybrid RRF (reciprocal rank fusion) merges the text ranking and the vector ranking into a single ordered result.

Reciprocal rank fusion: two independent rankings, one keyword and one semantic, combined into a single answer.

The payoff is measurable. On SIFT 1M (128-dim, recall ≈ 0.99) a single query sustains ~356 qps, and the index is 14–21× smaller than the HNSW indexes it’s measured against — small enough to sit in a fraction of the RAM, right next to your SQL. Under concurrent load it holds around ~700 qps on eight cores. And the build is parallel and streaming: the dataset is never fully materialised in memory, so it scales to tens of millions of vectors on a modest box rather than demanding one sized for the whole corpus at once.

Same recall, a fraction of the footprint — vectors that fit in RAM next to your SQL.

Why one file matters

Put together, this is the whole argument for Arkeion in miniature. SQL, full-text and vectors are one file, not three systems: no indexing pipeline to run, no service to keep in sync, no copies of your data leaving the boundary you control. Every part of it is versioned, so search can travel in time; every part is hash-chained, so verify() can prove the whole thing is intact. Search stopped being something you bolt on and became something the database simply is.

See the benchmarks for the full numbers, honestly reported — including the places where a dedicated engine still wins.