It started as a plain "PDF finder." A handful of e-books, an upload form, full-text search, nothing more was planned. But collections grow, and at some point an archive gets so big you can no longer keep track of it. Full-text search only found things when I already knew the exact words. If I ask "How does quantum entanglement work?" but the book says "entangled states," the search finds nothing.
So what's missing isn't a better search over words. What's missing is a search over meaning.
That realization comes quickly. The interesting part is the road from there to a system that answers a question in natural language, with citations, from my own documents, and entirely on my own machine. This post is about that road, and about the design decisions where it was actually decided.
The idea: don't search, answer
The technique behind this is RAG, short for Retrieval-Augmented Generation. Put simply, you pull the relevant passages out of your own documents and hand them to a language model as context. The model composes an answer from them and points back to the sources.
The appeal lives in the word "augmented." The model shouldn't confabulate from its murky training memory, but from what's actually written in my books. So not hallucination, but grounded answers. And because I wanted to run the whole thing locally with Ollama, not a single byte leaves the machine. No cloud service, no API bills. Flip a config value and it runs on OpenAI instead, more on that later.
The stack in brief:
- Elixir / Phoenix as an umbrella, with upload, search engine and web UI.
- PostgreSQL + pgvector, so vectors and full text in one database.
- Oban for background jobs (embedding, rendering).
- Phoenix LiveView for a chat UI with token streaming.
But the real work doesn't happen in the stack, it happens in the pipeline. At query time, meaning per question, it looks like this:
 |
| The RAG pipeline at query time: question, query rewriting, hybrid retrieval from semantic and full-text search, RRF fusion, LLM re-ranking, answer |
The foundation: hybrid retrieval
A RAG system is never better than what it can find. Generation is the easy part. The real work is in retrieval.
The obvious route is pure vector search: embed the question, chase the nearest text pieces in vector space. That's strong on meaning and weak on anything exact, so on proper names, codes, rare technical terms or version numbers. Pure full-text search is the mirror image, pedantic about wording and blind to meaning.
You don't have to choose. Both run in parallel:
- semantic, via cosine distance over the pgvector embeddings,
- full-text, via PostgreSQL's
websearch_to_tsquery / ts_rank.
Each side returns its own ranked list. They are fused with Reciprocal Rank Fusion (RRF), a pleasantly undramatic method that combines ranks rather than raw scores and thereby avoids the fragile job of normalizing two completely different score scales. A diversity cap of at most two hits per book then keeps a single chatty document from hijacking the whole result list.
Two small details bought surprisingly much recall. The HNSW index runs with a raised ef_search (100 instead of the default of about 40), because the default was silently capping the number of real candidates. And both signals run concurrently as parallel DB queries, so latency is that of the slower query rather than the sum. Embeddings are multilingual (bge-m3, 1024 dimensions), so a German question can find an English passage.
That's the foundation. Everything else is a layer on top. What's interesting is how independent those layers stay from one another.
Two layers at query time
The first is history-aware query rewriting. In a chat you ask follow-up questions. "And how does that work?" is meaningless without the conversation, and embedded it finds nothing. So before retrieval, the LLM rewrites the question against the history into a standalone search query. "And how does that work?" becomes "How does quantum entanglement work?".
The trick is the separation. Rewriting happens only for retrieval. The answer itself is still generated from the original question plus history, so it stays natural. On the first turn, when there is no history, or on an error, the raw question simply stands. Rewriting can never hurt, only help.
# Common.RAG.QueryRewriter
def rewrite(question, []), do: question # first turn: no LLM call
def rewrite(question, history) do
case Provider.chat(messages(question, history), []) do
{:ok, rewritten} -> sanitize(rewritten, question)
{:error, _} -> question # fallback: raw question
end
end
The second is LLM-based re-ranking. Fusion returns good candidates, but the order is only roughly by relevance. The textbook move would be a cross-encoder, a specialized model that reads question and passage together and emits a relevance score. Precise, but neither Ollama nor OpenAI offers a rerank endpoint. A real cross-encoder would therefore mean either a third cloud service with a new API key, or heavy local infrastructure, say a model of about 1 GB via Bumblebee/Nx.
I went with the pragmatic variant: a listwise re-ranker over the chat/2 I already have. The LLM gets the question plus numbered candidates and returns an order sorted by relevance. No new service, no new key, and it runs on Ollama and OpenAI right away.
# Common.RAG.Reranker, simplified
def rerank(question, chunks, k) do
case Provider.chat(messages(question, chunks), json: true) do
{:ok, response} -> chunks |> reorder(response) |> Enum.take(k)
{:error, _} -> Enum.take(chunks, k) # fallback: input order
end
end
For it, a larger pool of 20 candidates is fetched and sorted down to the best k (8). Unmentioned candidates are appended at the end, invalid answers fall cleanly back to the input order. That way re-ranking is never worse than no re-ranking.
Is it as good as a cross-encoder? No. But for a local, swappable stack it's the best compromise between quality and simplicity. And because the re-ranker sits behind a clear function, the engine can later be swapped for a real cross-encoder without touching a single caller.
The narrow seam: new sources, almost for free
Here's the design decision I'm proudest of. It wasn't really a deliberate decision, more an observation.
Knowledge doesn't live only in PDFs. It lives in blog posts, in doc pages, in Markdown READMEs. I wanted to paste a URL and then search and query it exactly like a PDF. The question was how deep I'd have to cut into the system for that.
The answer: not at all. Everything from the content field onward is already format-agnostic. The full-text index is a generated column over content, the chunker reads only content, and retrieval, re-ranking and chat know nothing but chunks plus an ebook_id. To ingest a web page, I had to change nothing in search, embeddings or RAG. It took a new ingestion source that fills content and title, plus a source_type discriminator.
For web pages the work is in the fetching. Many pages render their content only via JavaScript, so a plain HTTP request returns an empty shell. A headless Chrome renders the page and hands back the finished DOM, from which Floki pulls title and body text. That runs as an Oban job, because networking and rendering are slow and flaky. For Markdown even that is unnecessary, because the file's contents already are the content. Nothing to extract, nothing to render.
The simpler the source format, the more visible the seam. For Markdown, "wire up a new source" shrinks to three moves: store the file, set content to the file text, guess a title. Search, embeddings, re-ranking and chat notice the new source not at all.
This narrow seam has an honest flip side, because it carries only a single string. When page-precise citations came up later, that didn't help much. "It's in book X" isn't worth much across 300 pages, and structured page metadata doesn't fit through the seam. The fix wasn't to widen the seam but to carry the page boundary as a convention inside the string. pdftotext has always separated pages with a form feed (\f). The information was there all along, just unused.
# Common.Embeddings.Chunker: chunk a PDF page by page
def chunk_pages(content, opts \\ []) when is_binary(content) do
content
|> String.split("\f") # pdftotext separates pages with a form feed
|> Enum.with_index(1) # 1-based page number
|> Enum.flat_map(fn {page_text, page} ->
page_text |> chunk(opts) |> Enum.map(&%{page: page, content: &1})
end)
end
A plain re-embed made the existing corpus citable, with no re-extraction at all.
An honest dead end
A report that only tells the wins isn't one. Alongside the chunk approach I built a second retrieval path, the so-called IdeaBlocks. Instead of raw text pieces, you have an LLM distill question/answer blocks out of each document and deduplicate them across the corpus. The theory behind it: denoised, self-contained units of knowledge and therefore more precise hits.
Practice was sobering. Ingestion was expensive, because it needed several LLM calls per document. It lost information, since whatever the LLM doesn't recognize as an "idea" is gone on that path, while chunks preserve everything. And the retrieved text was a paraphrase rather than the source, so it was poor for faithful citation. In the end, hybrid fusion, re-ranking and HyDE over the full chunk corpus closed exactly the gap IdeaBlocks was meant to close, only more cheaply and only at query time.
Rather than clinging to it out of sunk-cost thinking, I built a small A/B tool that sent the same questions through both paths. The result was unambiguous, and the consequence was deletion. IdeaBlocks came out entirely, along with its pipeline, schema, retrieval path and UI toggle. Chunks is the only path today.
A tested hypothesis that fails isn't a failure, as long as you measure the failure and act on it. And sometimes acting on it means deleting the code again.
What held up
Boiled down to a few principles:
- Retrieval improvements are independent of ingestion. Rewriting, re-ranking and HyDE act at query time, so you roll them out without reprocessing the corpus.
- Every step has a fallback. No LLM call may make search worse, and on error the safe default holds (raw question, input order).
- Build against an abstraction, not a vendor. All AI operations go through a narrow behaviour (
embed, chat, chat_stream) with two implementations, Ollama and OpenAI. Rewriting, re-ranking and HyDE needed no new endpoint, they all build on chat/2.
- Measure instead of assume. A/B comparisons make design questions empirically decidable.
- A narrow, format-neutral seam. Because everything past
content is format-agnostic, a new source costs almost nothing. It's also a limit for structured metadata, which you then deliberately carry through as a convention.
A "PDF finder" has become a retrieval engine that knows three source formats, cites down to the page, and runs locally as well as in the cloud, without the core pipeline ever knowing where the text came from. To me that's the real punchline: the exciting extensions were almost never the ones that required cutting deepest into the system.
Kommentare